Conversational digital picture frame

ABSTRACT

A method and system for automated routing of pictures taken on mobile electronic devices to a digital picture frame including a camera, microphone, and speaker integrated with the frame, and a network connection module allowing the frame for direct contact and upload of photos from electronic devices or from photo collections of community members. Clustering photos by content is used to improve display and to respond to photo viewer desires. Trends or patterns can be detected from the photo collections and that information used for various purposes beyond photo display. The frame includes a conversational intelligence that provides a verbal communication with a viewer, such as for determining an identity or preferences of the frame viewer, determining photos to display for the viewer, discussing displayed photos with the viewer, or telling stories to the viewer based upon photo content.

FIELD OF THE INVENTION

This invention relates generally to a digital picture frame and, moreparticularly, to methods and systems for processing photos for a digitalpicture frame that is a Mobile Positional Social Media (MPSM) outputdevice that automatically provides personalized photo viewing inautomated combination with a proximate or remote electronic deviceand/or an MPSM account.

BACKGROUND OF THE INVENTION

Existing digital picture frames synchronize with popular social mediaplatforms, such as Facebook™ and Instagram™, using, among othertechniques, a WiFi™ interconnection. However, multiple steps may berequired for such synchronization. These manual steps can be difficultto do if one wants to change to a different social media account or todirectly retrieve from a picture capture or storage device, such aswhile entertaining guests. There is a continuing need for improvementsfor implementing displays such as home digital picture frames.

SUMMARY OF THE INVENTION

The present invention provides a digital picture frame thatautomatically detects viewers and/or viewers' electronic devices, andautomatically displays photos or videos relevant to the detectedviewers. The relevant photos or videos are obtained upon detection fromthe electronic device and/or social media accounts of the detectedviewers. The photos or videos can also be automatically obtained fromdevices, frames, stored photo collections, or social media accounts ofone or more community members of the detected viewer(s), such as photostaken and posted by a friend or relative of the detected viewer. Thedevice and method of this invention provides for improved and updatedphoto display based upon current photos and/or social media updates, andthus relevant to the viewer. The device of this invention can be used asan MPSM device to display any MPSM information, as desired, includingbut not limited to MPSM information obtained according to methodsdescribed herein.

A general object of the invention can be attained, at least in part,through a digital picture frame including a digital display mountedwithin a frame, a camera connected to the frame, a network connectionmodule adapted to connect to an electronic device, photo collection,and/or social media account of a viewer viewing the digital pictureframe and to receive photos stored within the electronic device and/orthe social media account, and an automated display module adapted toautomatically change photos displayed on the digital display to photosfrom the electronic device and/or social media account of the viewer,upon automatic detection with the camera of the viewer.

The invention includes a method of displaying photos on electronicdevices with a network connection, including: automatically obtaining adigital photo from a first electronic device; automatically identifyinga content of the digital photo; automatically identifying a targetdisplay electronic device as a function of the content; andautomatically routing the digital photo over the network connection tothe target display electronic device for display. The first electronicdevice is controlled by a first community member and the target displaydevice, such as a mobile device or a digital picture frame, can becontrolled by a second community member. The target display electronicdevice desirably includes predetermined content constraints, and thedigital photo is routed and/or displayed if the photo meets the contentconstraints.

The invention further includes a method of displaying photos onelectronic devices with a network connection via: automaticallyobtaining digital photos on a first electronic device; automaticallyidentifying a content of each of the digital photos; automaticallymatching the content of a first of the digital photos to a predeterminedcontent constraint of a second electronic device; and automaticallyrouting the first of the digital photos to the second electronic deviceover the network connection for display on the second electronic device.The method can further include automatically processing the digitalphotos by at least one of image enhancement, content filtering, and/orrepetition filtering. The content filtering can include identifying theplace, person, and/or activity, etc. of the photo, which can then becompared to preferred content (e.g., a stored metadata identificationset) of the potential target/receiving frames.

In presently preferred embodiments of this invention, where the contentof the first of the digital photos matches a predetermined contentconstraint of a plurality of community members, the first of the digitalphotos is automatically routed to electronic devices of each of thematching community members. The invention further includes a method ofdisplaying photos on electronic devices with a network connection,including: automatically obtaining digital photos on a first electronicdevice; automatically identifying a content of each of the digitalphotos; automatically matching the content of each of a plurality of thedigital photos to a predetermined content constraint of a correspondingelectronic device of one or more community members; and automaticallyrouting each of the plurality of digital photos to the correspondingelectronic device as a function of the matching for display on thecorresponding electronic device.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes: automatically determining with the camera anidentity of a viewer of the digital picture frame; automaticallydetermining a viewer profile from the identity of the viewer; andautomatically changing photos shown on the digital display as a functionof the viewer profile upon detection of the viewer with the camera. Themethod can include automatically augmenting a photo display on thedigital display as a function of photo metadata and the viewer profile.In embodiments of this invention, the viewer profile is automaticallylearned and uploaded via an electronic device and/or a social mediaaccount of the viewer.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes automatically determining with the camera anidentity of a viewer of the digital picture frame, automaticallyconnecting to a picture capture or storage electronic device of theviewer over a network, such as a WiFi network, and automaticallydisplaying photos shown on the digital display obtained from theelectronic device over the network connection.

The device and method of this invention can automatically display aslideshow of photos relevant to one or more display viewers as afunction of photo context selected from time taken, photo location,and/or photo content. The device can automatically access, using thenetwork module, at least one electronic device, digital photocollection, and/or social media account of: the viewer and/or at leastone social media community member of the viewer. Photos from the one ormore electronic devices, photo collections, or social media accounts aredesirably automatically loaded and displayed. In embodiments of thisinvention, obtained photos are automatically sequenced on the digitaldisplay as a function of profiling traits selected from chronologicalorder, photo location, photo activity, and/or community member.

The device and method can detect and recognize two or more different andsimultaneous, or otherwise present, persons viewing the digital pictureframe, and automatically change the photos displayed on the digitaldisplay to photos uploaded from an electronic device and/or a socialmedia account of each of two or more persons viewing the digital pictureframe. The photos from the electronic device or social media account ofeach of the two or more persons can include shared activity photos fromactivities shared by the two or more persons. The invention includesembodiments where a server computer automatically determines sharedactivity photos from the electronic device and/or social media accountof the each of the two or more persons as a function of contextinformation automatically associated with the shared activity photos bythe same or different server computer. The context information of eachof the shared activity photos can include, but is not limited to, aphoto location, a photo activity, and/or a present community member.

The invention still further comprehends a display device with one ormore of: automatic detection for configuration, namely theself-straightening or self-positioning on a vertical (e.g., wall) and/ora horizontal (e.g., table) platform; the automatic detection of viewingconditions, namely, based on the distance of the viewer, enlarging orcontracting the picture; based on the lighting, increasing or decreasingthe lighting intensity. The invention also comprehends a display devicethat: via face recognition, automatically detects the viewer and targetsthe photos for her or him based on their potentially learned profile;via the automatic identification and tagging of the location, activity,and community member involvement on a per photo or video, indicating thewith whom, where, when and what was being done when the picture wastaken as well as any other associated metadata; and/or providing a“story telling” capability. By storytelling, chronological stories,optionally simultaneously displayed on a split screen, are grouped by:purely time, namely in sequential or reverse chronological ordering;location, namely a traversal of sites on a location based trip;activity, namely in chronological ordering of a given or similar set ofactivities; community member involvement, namely a pictorial interactionwith community members, potentially segmented by particular communitymember or members; or any other profiling traits of a recognized userthat can be used to cluster or segment photos or videos for automaticstory telling.

The invention includes a method of displaying photos on a digitalpicture frame and including: automatically determining with the cameraan identity of a viewer of the digital picture frame; automaticallyconnecting to one or more digital photo collections over a network; andautomatically displaying on the digital display photos obtained from thedigital photo collection over the network connection. The digital photocollection(s) can be associated with a second digital picture frameand/or an electronic device of a community member of the viewer. Thedigital picture frame has access to the digital photo collection(s) anda server computer automatically sends the photos to the frame throughthe network connection module. The server or frame desirablyautomatically crops and aligns the photos to fit the digital display,such as by automatically scaling and realigning the photos to an aspectratio of the digital display.

Additional automated processing can include image enhancement, contentfiltering, or repetition filtering. The photos can be filtered by imagequality and content quality. The digital picture frame can include acontent restriction filter module including adjustable contentrestraints controlled by a frame administrator, for implementing contentrestrictions of the viewer obtained from a viewer profile over thenetwork. The frame administrator and/or the viewer can have or setpre-established filtering parameters selected from image quality and/orcontent quality, for use in filtering the digital photo collection.

In embodiments of this invention, the device and method includeautomatically detecting via the camera a physical response to one of thedisplayed photos, and automatically assigning an interest level grade tothe one of the displayed photos using the physical response. Thephysical response can be a body movement or a facial expression. Theinterest level grade can be stored in a viewer profile for the viewer,and used for displaying photos for the viewer as a function of theinterest level grade. The physical response, positive or negative, canbe used to identify and suggest other related photos to the viewer, suchas for a replacement photo related to the same subject matter.

In embodiments of this invention, the device and method includeautomated ranking of the photos to be circulated on the frame, such asfor automatically determining an order of presentation, a number ofdisplay repetitions, and/or a time period of display for each of thephotos as function of the ranking. The ranking can be based at least inpart on a physical feedback response of the viewer to one or more of thephotos, or based upon a picture quality and/or content. The photos canbe clustered according to the rankings into groups for display orsharing.

The invention further includes a method and device for displaying photoson a digital picture frame that includes: automatically clusteringphotos of one or more digital photo collections for display to theviewer, and automatically displaying on the digital display the photosfrom the digital photo collection as a function of the clustering. Theclusters are groups of photos to form photo collections of relatedcontent, such as selected from people, things, places, activities, orcombinations thereof, and displaying the photo collections. The photosare desirably clustered by a photo content, which can be identified orconfirmed by photo metadata. Clustering of photos with a preselectedphotographed person can be performed by automatically identifying andclustering the photographed person by facial features. Clusteringaccording to this invention is particularly useful for photos taken overa multi-year timeframe, and grouping photos by related photo contentacross the timeframe.

Embodiments of this invention further include additional uses for thevoluminous amounts of photos that flow through the digital picture framesystems of this invention. The invention includes a method of processinga digital photo collection for display on a digital picture frame. Themethod includes: automatically extracting content features from photosof the digital photo collection and/or automatically extracting photoimage features from photos of the digital photo collection; creatingmetadata tags for the photos as a function of the extracted contentfeatures and/or the extracted photo image features; storing taggedphotos in a database; and mining correlations within the metadata tags.The mining generally includes determining relationships betweendifferent metadata tags of a plurality of the tagged photos.

In particular embodiments, the tagged photos are clustered (e.g., by thetags) into a plurality of sub-clusters, each for a corresponding commondetected extracted content feature and/or extracted photo image feature.The mining can include comparing sub-clusters of photo (e.g., via thetags) across more than one digital photo collection over a network andmining correlations from the more than one digital photo collection.

The extracted content features can be, for example, a location, aseason, a weather content, a time of day, a date, an activity content,and/or one or more attributes of a person of the photos. The extractedphoto image features can be, for example, a light intensity or a colorwithin the photos, or other camera metadata added to the digital photofile. As a further example, the extracted photo image features can be aclothing color of the person of the photos.

In embodiments of the invention, mining correlations includes rulemining the metadata tags, such as by association rule mining bygenerating a set of association rules or implications with a confidence,and support for the set. The metadata tags can be stored in amulti-dimensional dataset, and the mining correlations can includeperforming slicing operations and/or dicing operations on themulti-dimensional dataset. Exemplary mining correlations includedetermining popular locations, clothing items, colors, and/or activitieswithin the plurality of the tagged photos, for a predetermined timeperiod and/or a predetermined age group.

In embodiments of this invention, the frame (and/or the interactionapplication of the frame system loaded onto user mobile devices)includes a conversational intelligence component for conversation withone or more viewers of the frame. The conversation includes a verbalinteraction between or among the frame and one or more viewing persons,preferably about one or more displayed photos or to determine whatphotos to display. The verbal interaction can be used, withoutlimitation, for entertainment purposes (e.g., general conversation,telling jokes, telling stories, or providing educational orinstructional information), for providing information to the viewer(e.g., about the photos or content in the photos), for determining orimproving a mood or condition of a viewer, as a voice assistant, fordetermining search parameters for determining photo displays, andcombinations thereof.

As used herein, a “verbal interaction” of a frame device refers to anexchange of words, phrases, sentences, and/or a visual ASL display witha viewer, where the frame initiates, receives and/or responds verbally,using a speaker and microphone, or ASL via camera and display, to aframe viewer. The verbal interaction is generally a back-and-forthconversational exchange between the frame and viewer that imitates humaninteractions, recognizing speech and/or text inputs (and translatingmeanings across various languages as needed), such as using any suitableconversational artificial intelligence, generally based upon naturallanguage processing and generation, and machine learning. Naturallanguage processing (NLP) generally refers to the ability of the frameor frame system to ‘understand’ the verbal or written content producedby the viewer or other user. Natural language generation (NLG) generallyrefers to the frame or frame system's ability to create content ineither written (e.g., on the frame display) or spoken language (e.g.,through the speakers or Bluetooth® headphones, etc.) so that it can beunderstood by the viewer or another user.

Embodiments of this invention include a method of displaying photos on adigital picture frame or frame system, as described herein, such asincluding a digital display mounted within a frame, a microphone andspeaker connected to the frame, and a network connection module. Themethod includes: automatically providing a verbal interaction with aviewer of the digital picture frame; automatically determining a searchparameter from the verbal interaction; automatically connecting to adigital photo collection over a network; automatically searching thedigital photo collection for digital photos matching the searchparameter; and automatically displaying on the digital display matchingphotos obtained from the digital photo collection over the networkconnection. The verbal interaction can also include automaticallygenerated questions to the viewer about any displayed photo.

Methods of this invention additionally or alternatively include stepsof: receiving a story request from a viewer of the digital pictureframe; automatically searching for and/or displaying one or more digitalphotos of the digital photo collection matching the story request; andautomatically generating a unique narrative (e.g., through the speaker)matching the story request and the one or more digital photos, whereinthe unique narrative is guided by displayed image content, photometadata, or settings provided by the viewer. As used herein, a “uniquenarrative” is intended to refer to or include an automatically generatedstory or storyline (fiction and/or partially or fully non-fiction), suchas including one or more characters, setting(s), and a plot, preferablybased at least partially on or corresponding at least partially to aphoto content (e.g., person, location, etc.) of a displayed photo and/orphotos to be displayed with or during the automatically generated andspoken story.

The invention further includes methods including steps of: automaticallyclustering original photos within a digital photo collection into aplurality of sub-clusters, each for a corresponding common detectedcontent in the original photos selected from a person, a thing, a place,an activity, or combinations thereof, wherein each of the plurality ofsub-clusters is formed for the corresponding common detected content ofinterest at a similar time and/or age; extracting one or more photofeatures from the corresponding common detected content within the eachof the sub-clusters of the original photos to form a clusterrepresentation, wherein the photo features are predetermined as havingthe limited change over a timeframe; comparing the clusterrepresentation of the each of the sub-clusters to new photos added tothe digital photo collection, to determine a corresponding sub-clusterfor each of the new photos; linking the each of the new photos to thecorresponding sub-cluster for display; automatically displaying aslideshow on the digital display of the original photos and the newphotos from the digital photo collection as a function of theclustering; automatically providing a verbal interaction with a viewerof the digital picture frame via the microphone and speaker;automatically determining a search parameter from the verbalinteraction; automatically connecting to the digital photo collection;automatically searching the digital photo collection for digital photosmatching the search parameter; and automatically displaying on thedigital display matching photos obtained from the digital photocollection over the network connection.

The invention further includes methods including steps of: automaticallyclustering original photos of a digital photo collection for display tothe viewer, wherein the clustering comprises: preparing a plurality ofsub-clusters, each for a corresponding common detected content in theoriginal photos selected from a person, a thing, a place, an activity,or combinations thereof; extracting one or more photo features from thecorresponding common detected content within the each of thesub-clusters of original photos to form a cluster representation,wherein the one or more photo features are predetermined as having alimited change over a timeframe; comparing the cluster representation ofthe each of the sub-clusters to new photos in the digital photoscollection or in a second digital photo collection, to determine acorresponding sub-cluster for each of the new photos; linking the eachof the new photos to the corresponding sub-cluster for display;automatically displaying a slideshow on the digital display of theoriginal photos and the new photos from the digital photo collection asa function of the clustering; automatically verbally interacting with aviewer of the digital picture frame via the microphone and speaker;automatically determining a search parameter from the verbalinteracting; automatically connecting to the digital photo collectionover the network; automatically searching the digital photo collectionfor digital photos matching the search parameter; and automaticallydisplaying on the digital display matching photos obtained from thedigital photo collection over the network connection.

In embodiments, the method or system includes automatically determiningwith automated questions an identity of a viewer of the digital pictureframe. The verbal interaction desirably includes a plurality ofautomated back-and-forth conversational iterations with the viewer toestablish the identity of the user and/or the search parameter.

The search parameter can include a person, an activity, a location, orcombinations thereof, identified from keywords parsed and extracted inthe verbal exchange. The search can be performed on content clusteredphotographs of the digital photo collection, clustered according to theperson, the thing, the place, the activity, or combinations thereof,and/or performed on metadata associated with photos in the digital photocollection. Clustering can be performed as described herein. Metadata ofthe original and any new photos can be used to identify or confirm theoriginal or new photos in the sub-clusters, the metadata includingphotograph dates.

In embodiments of this invention, the frame system automatically detectsvia the microphone and/or the camera a vocal or physical response fromthe viewer to one of the displayed photos, and adjusts the verbalinteraction as a function of the vocal or physical response.

Embodiments of this invention include a method implemented via a frameincluding steps of: automatically clustering original photos within adigital photo collection into a plurality of sub-clusters, each for acorresponding common detected content in the original photos selectedfrom a person, a thing, a place, an activity, or combinations thereof,wherein each of the plurality of sub-clusters is formed for thecorresponding common detected content of interest at a similar timeand/or age; extracting one or more photo features from the correspondingcommon detected content within the each of the sub-clusters of theoriginal photos to form a cluster representation, wherein the photofeatures are predetermined as having the limited change over atimeframe; comparing the cluster representation of the each of thesub-clusters to new photos added to the digital photo collection, todetermine a corresponding sub-cluster for each of the new photos;linking the each of the new photos to the corresponding sub-cluster fordisplay; automatically displaying a slideshow on the digital display ofthe original photos and the new photos from the digital photo collectionas a function of the clustering; receiving a story request from a viewerof the digital picture frame; automatically searching for and/ordisplaying one or more digital photos of the digital photo collectionmatching the story request; and automatically generating a uniquenarrative through the speaker matching the story request and the one ormore digital photos, wherein the unique narrative is guided by displayedimage content, photo metadata, or settings provided by the viewer.

The story request can be in response to a currently displayed photo, orcan trigger a photo or slideshow displayed in response to the storyrequest. In embodiments a slideshow of photos related to the uniquenarrative is played along with the unique narrative.

In embodiments of this invention, the displayed photo is annotated forthe narrative with overlaid text or images related to the uniquenarrative. The frame can desirably also automatically determine and usea spoken dialect for the unique narrative that corresponds to a person,location, or activity of the one or more photos.

In embodiments, the frame receives/detects responsive vocal utterancesor voice prompts from the viewer during the unique narrative, andautomatically adjusts a storyline of the unique narrative in response tothe responsive vocal utterances or voice prompts. The frame can thenautomatically search for and display a further photo relevant to theadjusted storyline. Settings provided by the viewer can also include adesired sentiment or motivation for the unique narrative.

In embodiments, automatically generating the unique narrative includesanalyzing metadata for the one or more digital photos.

The unique narrative can include an initial verbal interactiveconversation with or about a content of the displayed photo. The verbalinteraction is desirably a plurality of automated back-and-forthconversational iterations between the frame and a viewer to establishthe identity of the user and/or a desired narrative type. The uniquenarrative then includes a person, an activity, a location, orcombinations thereof, identified from keywords detected in the verbalinteractive conversation.

The invention alternatively or additionally includes a method havingsteps of: automatically extracting content features from photos of thedigital photo collection; automatically extracting photo image featuresfrom photos of the digital photo collection; creating metadata tags forthe photos as a function of the extracted content features and theextracted photo image features; storing tagged photos in a database;automatically providing a verbal interaction with a viewer of thedigital picture frame; automatically determining a search parameter fromthe verbal interaction; automatically connecting to the digital photocollection over a network; automatically searching the digital photocollection for metadata tags and digital photos matching the searchparameter; and automatically displaying on the digital display matchingphotos obtained from the digital photo collection over the networkconnection.

Embodiments include a method having steps of: automatically extractingcontent features from photos of the digital photo collection;automatically extracting photo image features from photos of the digitalphoto collection; creating metadata tags for the photos as a function ofthe extracted content features and the extracted photo image features;storing tagged photos in a database; automatically displaying on thedigital display a photo obtained from the digital photo collection; andautomatically providing a verbal interaction about the displayed photobetween the digital picture frame and a viewer of the digital pictureframe.

Relationships determined between different metadata tags of a pluralityof the tagged photos can be used in the verbal interaction.

In embodiments, the method includes mining correlations within themetadata tags. The mining correlations can include rule mining themetadata tags. The rule mining can include association rule mining bygenerating a set of association rules or implications with a confidence,and support for the set. The mining correlations comprises determiningpopular locations, clothing items, and/or activities within theplurality of the tagged photos, for use in the verbal interaction withthe viewer.

In embodiments, the verbal interaction includes a plurality of automatedback-and-forth conversational iterations, and the mining correlationsincludes determining popular locations, clothing items, and/oractivities within the plurality of the tagged photos, for use as contentin conversational iterations of the verbal interaction with the viewer.

Embodiments of the invention include a method of displaying a digitalphoto collection on a digital picture frame, including steps of:automatically providing a verbal interaction with a viewer of thedigital picture frame; automatically determining a search parameter fromthe verbal interaction; automatically connecting to the digital photocollection over a network; automatically searching the digital photocollection for metadata tags and digital photos matching the searchparameter; automatically displaying on the digital display a matchingphoto obtained from the digital photo collection over the networkconnection; receiving a story request for the matching photo from theviewer of the digital picture frame; and automatically generating aunique narrative through the speaker matching the story request and thedigital photo, wherein the unique narrative is guided by displayed imagecontent, photo metadata, settings provided by the viewer. The method canfurther include steps of: receiving responsive vocal utterances or voiceprompts from the viewer during the unique narrative; optionally alsoautomatically detecting a sentiment or mood of the viewer from the vocalutterances or voice prompts; and automatically adjusting a storyline ofthe unique narrative in response to the responsive vocal utterances orvoice prompts, and/or the detected sentiment or mood. The uniquenarrative can be automatically developed from information extracted fromcontent clustered photographs of the digital photo collection, clusteredaccording to photo content of persons, activities, locations, orcombinations thereof, and/or clustered photo features extracted from thephotos in the digital photo collection.

The invention further includes a digital picture display systemincluding a digital photo frame with a digital display, a networkconnection module adapted to connect to a user electronic device and toreceive photos stored within the user electronic device, an automateddisplay module adapted to automatically display photos and change photosdisplayed on the digital display from the user electronic device, amicrophone in combination with the automated display module and adaptedto receive spoken communications, and a speaker in combination with theautomated display module and adapted to produce audio communications.The digital picture display system al so includes an interactionapplication stored on the user electronic device, the user electronicdevice including a digital photo collection, and the interactionapplication including a plurality of automatically created digital photoclusters according to photo content from the digital photo collection.The digital photo frame further includes a conversational agent moduleconfigured to provide an automated verbal interaction with a viewer ofthe digital picture frame. The conversational agent module determines asearch parameter from the verbal interaction and the search parameterinitiates a search of the digital photo clusters for metadata tags anddigital photos matching the search parameter. The network connectionmodule receives matching digital photos from one or more of the digitalphoto clusters for display on the digital display.

In embodiments, the conversational agent module is configured toautomatically determine with questions through the speaker an identityof the viewer of the digital picture frame. The verbal interactiondesirably includes a plurality of automated back-and-forth iterations ofconversational vocal utterances via the microphone and speaker toestablish an identity of the user and/or the search parameter.

The conversational agent module can include a story-tellingfunctionality or module configured to receive a story request from aviewer of the digital picture frame, automatically search for and/ordisplay one or more digital photos of the digital photo collectionmatching the story request, and automatically generate a uniquenarrative, such as discussed above, through the speaker matching thestory request and the one or more digital photos. The unique narrativeis desirably guided by displayed image content, photo metadata, and/orsettings provided by the viewer. The conversational agent module canreceive the story request from the viewer in response to a photo beingdisplayed on the digital display. The conversational agent module caninstruct the automated display module to show a slideshow of photosrelated to the unique narrative.

In embodiments, the conversational agent module is configured to detectand analyze responsive vocal utterances or voice prompts from the viewerduring the verbal interaction to automatically adjust the verbalinteraction or change a displayed photo in response to the responsivevocal utterances or voice prompts.

In embodiments, a further photo is obtained from the digital photocollection and the conversational agent module is configured toautomatically continue the verbal interaction with a viewer of thedigital picture frame about the displayed second photo.

In embodiments, a second interaction application is stored on a secondelectronic device of a community member. The second electronic deviceincludes a second digital photo collection, and the second interactionapplication includes a second plurality of automatically created digitalphoto clusters according to content from the second digital photocollection. The search parameter from the verbal interaction is searchedwithin the digital photo clusters of the second digital photo collectionfor metadata tags and digital photos matching the search parameter. Thenetwork connection module receives matching digital photos from thesecond digital photo collection for display on the digital display.

In embodiments, a facial recognition module is in combination with theconversational agent module and/or the automated display module. Theautomated display module automatically changes a photo display, aconversation, and/or a storyline (e.g., characters or plot) upondetection of a recognized viewer with the facial recognition module.

In embodiments, the conversational agent module or the automated displaymodule includes or obtains a viewer profile of the recognized viewer andautomatically changes the display as a function of the viewer profileand the corresponding electronic device upon detection of the recognizedviewer through the camera.

The digital picture frame of embodiments of this invention operates inconcert with a method, system, and apparatus, such as embodied in anMPSM or other software application, that automatically determines andshares a location and/or an activity of a user. The application learnsuser activity over time, with the learning based upon user locationsand/or context. The present invention generally provides methods andapplications for an MPSM that automatically understands and informs the“who, what, when, where, and/or how” of a user and the user's community.For example, who are the user and their community with, what are theuser and their community doing, where geographically are the user andtheir community, when are, and historically when were, the user andtheir community doing this, and/or how can users' behaviors be modified?The method includes automatic tagging of photos taken by the users withcorresponding context data (e.g., location, activity, people present)learned by the system for use in determining photos to display on thedigital picture frame of this invention as discussed above.

The invention includes a method of learning and sharing locations and/oractivities, and pictures generated therefrom, of a user participating ina social networking service. The method is executed by a softwareapplication stored and executed on one or more computers or dataprocessor systems, such as a mobile device (e.g., phone, tablet, orlaptop) and/or an application server (such as for connecting usercommunities). In one embodiment the method includes receiving userinformation about a destination, automatically associating the userinformation with the destination, and automatically sharing the userinformation in the social networking service upon further user arrivalsat the destination prior to receiving any additional user information.

The invention still further includes a computer server for providing atracking and/or social networking service, such as operating the methodsdiscussed herein. The computer server of one embodiment of thisinvention includes a tagging module configured to correlate userinformation and/or pictures thereof to a user destination or activity, adatabase module configured to store user information including userlocations and user activities at the user locations, and a communicationmodule configured to automatically share a user activity in the socialnetworking service upon further user arrivals at a corresponding one ofthe user locations. The computer system can also include an associationmodule configured to associate the user activity with the correspondingone of the user locations.

In embodiments of this invention, the system or application identifieslocations, and over time, automatically “checks-in” not only thelocations but what the locations imply in terms of potential activitiesof the user. That is, given a location and a user, the system desirablysuggests what activity or activities the user typically partakes at thatlocation. For example, if a user frequents a location in “Potomac,” thislocation might be identified as “parents' home.” Furthermore, at thishome, a variety of activities might be common such as: “visitingparents,” “drinking tea,” “eating lunch,” or “sampling wine.” Inembodiments of this invention, each time the user appears at thatlocation, based on context, defined by elements of or surrounding theactivity such as but not limited to time of day, day of week,immediately preceding activities, weather, surrounding people, etc., aset of likely occurring activities are identified. The user can beprompted with a list from which to select a subset of these activitiesor to identify a new activity. The invention can also include ranking auser's potential suggested activity based on context and presenting thatranked list to the user, or the user's community. The inventiongenerally provides a learning component that can allow the manual inputsto become automatic prompts, which can become automatically issuednotifications for the location based upon the context. The prompts canbe issued through any known format, such as an application alert on thedevice or a text message to the user. The invention also supports theuser changing activities for a given location at any time, and/or userimplemented delay of the notification of a user's location or activity.

The invention can include the incorporation or creation of usercommunities and sub-communities, with such communities andsub-communities sharing information. Embodiments of the inventioninclude automatically identifying a user's location and activity, anddesirably notifying that user's community of that user's location, aswell as sharing relevant photos from the user's community to the digitalpicture frame. Particular embodiments of this invention provide one ormore additional community functionalities including, without limitation,automatically identifying a user's activity and notifying that user'scommunity of that user's activity, commenting on user activity andlocation report by user or community—with multiple and multimediacomments supported, supporting the “liking” of user activity by thecommunity, supporting the user tagging of location, activity, or thepairing of location and identity—tagging can be textual or via anymultimedia means, correlating the individual user's activity with theongoing activity of others within the community, and/or correlating theindividual user's activity with the past activities of others within thecommunity, all useful in generating user profiles and tagging photos forautomated display on the digital picture frame of this invention.

One embodiment of this invention provides a method of and system forautomated determining of locations and/or activities of a userparticipating in a social networking service. The method is executed byan MPSM computer system and automatically determines a positionaldestination of a user, automatically deduces as user information alocation type and/or user activity of the positional destination, andautomatically shares the user information as instructed in the socialnetworking service. Deducing the user information is based upon contextinformation about the positional destination, desirably with minimal orno input by the user. The context can include, without limitation,time-dependent information, past and/or current associated userinformation, past user and/or community information about the location,and/or third party information. The context can be used to at leastreduce location types and/or user activities, for example, as a functionof the past location type and/or user activity of the positionaldestination for a given time period.

The invention further includes marketing methods for the frame thatmotivate multiple purchases. First, volume discounts are provided forthe simultaneous purchasing of multiple frames. Additionally, the timeof the most recent purchase is noted, and should another purchase by thesame customer occur within a predetermined, but possibly varying, amountof time, a discount is applied. The duration might be based on theseason, sales history, stock availability, customer identity, or anyother business driven rule.

Other objects and advantages will be apparent to those skilled in theart from the following detailed description taken in conjunction withthe appended claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIG. 2 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIGS. 3 and 4 illustrate a photo clustering according to one embodimentof this invention.

FIG. 5 shows an exemplary photo metadata tagging, according to oneembodiment of this invention.

FIGS. 6-8 illustrate mining a dataset of photo metatags, according toone embodiment of this invention.

FIG. 9 shows a representative area of a user according to an MPSM methodof one embodiment of this invention.

FIG. 10 illustrates geofences surrounding a current reading and itsimmediate neighbors according to one embodiment of this invention.

FIG. 11 illustrates the determination of a location via intersectingcircles according to one embodiment of this invention.

FIG. 12 illustrates the processing flow employed to identify an arrivalaccording to one embodiment of this invention.

FIG. 13 illustrates a digital picture frame interaction with a vieweraccording to one embodiment of this invention.

FIG. 14 illustrates a digital picture frame interaction with a vieweraccording to one embodiment of this invention.

FIG. 15 illustrates a digital picture frame interaction with a vieweraccording to one embodiment of this invention.

FIGS. 16-18 illustrate a digital picture frame interaction with a vieweraccording to one embodiment of this invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a digital picture frame that interactswith either a picture capture or storage device, referred to herein asan electronic device, running a digital picture frame application and/orsocial media to automatically display digital photographs from theelectronic device and/or a social media account of one or more personsviewing the picture frame. As used herein, the terms “picture”,“photograph”, or “photo” refer to or include digital still photographsor digital video, including a combination of few-second animationseither or both before and after a still frame, unless otherwise noted.The photos can be obtained by social media users and stored in theusers' social media accounts. When a social media user views the digitalpicture frame, the digital picture frame automatically recognizes thatuser and automatically loads photos relevant to that viewer from her orhis electronic device and/or social media account and/or from at leastone electronic device and/or social media account of her or his socialmedia community member(s). For example, relevant photos from a communitymember's social media account may include the viewer or be from a sharedactivity that the viewer attended with the community member. Automatedsocial media activity learning methods, as described herein, are used toautomatically determine relevancy of photos to one or more communitymembers. Community members may choose to make photos shared or privateto avoid personal photos being shared.

FIG. 1 illustrates a digital picture frame 20 according to oneembodiment of this invention. The digital picture frame 20 includes adigital display 22 mounted within an edge frame 24. The digital display22 can incorporate any suitable screen, such as, without limitation, anLED screen or a touch screen, as are known and commercially available. Acamera 25 is connected to and integrated in the edge frame 24. Thepicture frame 20 includes a network connection module that connects overa wired and/or wireless network to a social media server computer 26 andcan automatically load photos stored within one or more social mediaaccounts accessed through the server 26. The picture frame 20 furtherincludes an automated display module in combination with the networkconnection module and adapted to automatically change photos displayedon the digital display 22 to photos from an electronic device and/or asocial media account of the viewer and/or one or more community members,such as upon automatic detection with the camera of the viewer. Thedigital picture frame 20 desirably includes a microphone 27 incombination with the automated display module and/or network connectionmodule, such as for receiving spoken instructions from a viewer tochange the displayed photos.

The digital picture frame 20 further includes a leveling device 28connected to or otherwise integrated with the frame 24. The levelingdevice 28 automatically detects when the frame is not level. Properpositioning of the digital picture frame is important to achieving usersatisfaction. Auto detection of the positioning in comparison to theangle of the displayed photograph illustrated can be achieved using anyof the image alignment or realignment techniques known in the art. Inembodiments of this invention, the leveling device 28 can transmit ordisplay corrective measures for physical correction.

In another embodiment of the invention, the digital picture frame 20 isconfigured with a leveling device that includes one or more motors 23,such as paired attached with a corresponding wall bracket via gearing.The motors 23 can be incorporated in a wall hanging mechanism as shownin FIG. 1 , or otherwise implemented with respect to a frame stand. Upondetection of an unleveled viewing as determined by either the levelingdevice 28 or any other mechanism, the dual motors 23 activate andautomatically mechanically operate as needed to level the digitalpicture frame 20, such as a vertical adjustment on wall mountinghardware, via the adjustable hanging mechanism or bracket.

The digital picture frame 20 includes an automated display adjustmentdevice in detection combination with the camera 25. The automateddisplay adjustment device automatically detects frame environment and/orviewer position using the camera 25 and automatically adjusts settingsof the digital display 22. Augmented with the camera 25, the digitalpicture frame 20 supports a host of optional, additional capabilities.Using conventional, known in the art, image processing techniques,enhancement of the picture quality can be automatically made. Forexample, due to an automatically detected viewing angle or environmentalbrightness, picture lighting intensity and contrast can be altered. Thatis, depending on conditions, the brightness and contrast can beautomatically modified to provide better viewing. Additionally,depending on the distance and angle of the viewer, the size of the imagecan be enlarged, possibly focusing on the center or content rich area ofthe picture. Similarly, a close viewer may wish to have the photographor video in full scope, or maybe, even side-by-side with one or moreother, possibly related pictures or videos.

In one embodiment, as shown in FIG. 1 , the digital picture frame 20includes the capability to ‘daisy chain’ one or more additional frames20′. That is, all participating frames can be connected so as to portraya single story, either chronologically or thematically, allow for twoviewers, and/or provide a composite view of a single image. The frame 20can include a network connection 10, such as an audio-video outletand/or network outlet, to connect to a corresponding inlet 12 of thesecond frame 20′. The network connection between frames can be by wiredconnection (e.g., HDMI cord or Ethernet cable) or wireless communication(e.g., Bluetooth or WiFi) The frame 20 can optionally include a poweroutput, such as outlet 14 to receive a power cord 16 of the second frame20′, so as to require only a single power source.

An embodiment of a digital picture frame 20 further includes multiplepower options. In addition to conventional outlet and battery options,the frame can include at least one photoelectric cell. Inductive, orwireless, charging can also be incorporated.

The digital picture frame 20 includes a facial recognition module withany suitable facial recognition software, in combination with the camera25 and automated display module. Face recognition software can bedeployed to automatically detect the identity of the viewer(s). It isalso within the scope of this invention to identify the viewer via thelocation or proximity of their personal electronic devices such as butnot limited to their mobile phones or tablets. By identifying theviewer(s) the automated display module can then specially target, namelyselect, prioritize, and/or order the photos or videos displayed for theidentified viewer.

Identifying the viewer enables a variety of novel applications. Inembodiments of this invention, a viewer analysis module is combined witha facial recognition module to determine, for example, one of mood orhealth of the viewer upon detection of the viewer with the facialrecognition module. The automated display module can augment a photodisplay on the digital display as a function of the determined mood orheath of the viewer, such as by showing photos to increase the viewer'smood.

Longitudinal analysis of the viewer is possible. Such analysis supports,for example, health commentary. By analyzing the viewer's face, commentssuch as, but not limited to, “you seem to have gained weight”, “you looktired”, or “you seem pale; do you feel well?” can be announced to theviewer. Similarly, “mirror, mirror on the wall” remarks such as “youlook good today” can be stated, potentially elevating the viewer's mood.Displaying identified mood enhancing photographs, such as, for example,ones that include far away loved ones can likewise be shown to furtherelevate the viewer's mood. Identification of mood enhancing photographscan be automatically accomplished, for example, via the recognition of a“smiling face” or spoken words of the viewer when the picture waspreviously shown, and tagging of the photo.

Viewer identification also enables photo selection personalization. Aviewer personalization module can be operatively combined with thefacial recognition module, whereby the viewer personalization moduleautomatically identifies preferred photos or restricted photos as afunction of viewer information upon detection of the viewer with thefacial recognition module. Photos favored by the viewer (e.g.,grandchildren for the grandparent) can be selected, and converselyphotos containing, for example, ex-spouses or in-laws can be repressed.Likewise, identifying viewer characteristics, such as age, enables thelimiting of photos displayed. For example, children might not be shownphotos containing potentially inappropriate content, such as containingnudity, or particular activities or individuals, such as relatives thathave recently passed away. Thus, it is within the scope of thisinvention to limit or specifically select photographs to be displayedbased on relationships of the viewer with individuals in thephotographs, such as ex-spouses; characteristics of individuals shown inthe photographs, such as babies; characteristics of the viewer, such asage; activities illustrated in the photographs, such as smoking;locations displayed in photographs both physically described (e.g.,Chicago) or relatively described (e.g., viewer's place of residence); orbased on any characteristics that can be determined via image analysisor photograph metadata included. The filtration or selection ofphotographs based on criteria is determined by settings that can be set.

Additional specialty settings are included. Such setting place the framein nightlight mode where only a dim glow is illustrated; photo recordmode to simply record all in the line of vision; video chat mode withother devices; and “stranger in the house” mode where only limited orstock photos are shown to prevent the potential identification ofresidents.

The digital picture frame 20 can likewise be used as a viewer assistant.Multiple ailments or conditions, such as but not limited to Alzheimer's,strokes, and dementia, affect memory. Thus, the digital picture frame 20can be used to remind the viewer of the identity of family members andfriends as well as of occasions, locations, activities, etc., ofrelevance as the photos displayed are labeled both by content and/or bymetadata.

As previously discussed, the network connection module connects over anetwork connection to the social media server computer 26. It canlikewise connect either directly or indirectly to the mobile device 29′of any community member or interested party having an appropriatelyconfigured device, such as digital picture frame application specificsoftware. The connection need not be within a geographic proximity, andit is likewise within the scope of this invention that a geographicallyremote connection is supported via networks such as but not limited toas the Internet.

The social media server computer 26 can be any one or more computersthat implement any suitable social media platform, such as the MPSMplatform and automated learning steps described herein. The servercomputer 26 can obtain or have access to photos and learn user oractivity information associated with each photo from, for example,mobile devices 29 of a plurality of MPSM users or community members,such as according to methods described herein. In embodiments of thisinvention, the automated display module augments a photo display on thedigital display 22 as a function of photo metadata from the social mediaaccount stored on the server computer 26. The photos are automaticallydownloaded from the server 26 to the digital frame 20 upon, for example,automated detection of a corresponding vi ewer(s).

In embodiments of this invention, the photos can be changed according tothe learned bias of a viewer, expressed and stored as part of a userprofile at the server computer 26 and/or the digital picture frame 20.The picture frame 20, such as via the automated display module, includesor obtains the viewer profile of the viewer upon detection of the viewervia the camera. By establishing a viewing profile for each known orexpected viewer, e.g., if a viewer is or has a community member,individual or group user viewing preferences can be maintained. Aprofile can be established using any one or more of the many knownprofile techniques known in the art, including but not limited to,asking the viewer, learning based on previous user selections, and/orcorrelations with previously automatically learned, via MPSM methodsherein, locations, activities, and/or community member interactions.

In a preferred embodiment, viewer profiles are automatically learnedaccording to methods described herein, and uploaded via an electronicdevice and/or the social media account of a detected viewer and/or heror his community members. The automated display module desirablydisplays a slideshow of photos for a detected viewer that is uploadedfrom: an electronic device of the viewer, and/or a social media accountof the viewer or one or more social media community member of theviewer.

In the digital picture frame embodiments of this invention pictures orvideos displayed can be augmented with the automatically learned andidentified metadata corresponding to the pictures or videos. That is,the information learned, captured and/or displayed identifies thelocation, activity, and community member involvement on a per photobasis, indicating the with whom, where, when and what was being donewhen the picture or video was taken as well as any other associatedmetadata, such as the camera or video equipment used to capture thephoto or video. Slideshow of photos for a viewer can be assembled andshown in a display order automatically determined as a function of photoprofiling traits such as time (e.g., chronological order), photolocation, photo content (e.g., people or activity shown in the photo),and/or community member presence at the time/location that a photo wastaken.

The invention includes a method of displaying photos on a digitalpicture frame, such as shown in FIG. 1 . The digital picture frame ishung or set on a surface by a user, for explanation purpose, namedSusan. The picture frame automatically determines with the camera whenSusan is viewing, or is the primary viewer of, the picture frame. Thepicture frame automatically determines Susan's profile from herdetection. In embodiments of this invention, Susan's viewer profile isloaded from Susan's electronic device and/or social media account, andmay be supplemented by additional information gathered via the pictureframe itself, such as face recognition, account access information,and/or manual entry (e.g., picture frame display preferences). Theviewer profile is desirably created and continually updatedautomatically according to the automated learning methods describedherein. The viewer profile can include information such as, withoutlimitation, family information, community member information, and/oruser favorites (e.g., favorite locations, activities, colors, flowers,animals, etc.), learned through the social media account.

When the camera and coordinated software determines Susan is viewing thepicture frame, e.g., either directly or merely in the vicinity, thepicture frame automatically displays photos shown on the digital displayas a function of the viewer profile information. The photos displayedare automatically augmented or picked as a function of comparing photometadata and the viewer profile. In embodiments of this invention, thepicture frame automatically displays for Susan a slideshow of photos(still and/or videos) relevant to Susan as a function of photo context.The photo context can be selected from the time the photo was taken(e.g., all photos from July 2015), photo location (e.g., all photostaken in Chicago), and/or photo content (e.g., all photos of vacationsor involving her sister and/or her friend Mary). The photos can beimported primarily from her electronic device and/or social media, andoptionally supplemented from photos on a recordable medium of the frame(e.g., hard drive or inserted flash drive or memory card) or photosobtained from a third party web site.

It is within the scope of this invention to prompt the user, possiblyvia any communication mechanism including but not limited to text andvoice commands, to determine the exact wishes of the user. If thedigital picture frame is additionally equipped with a microphone 27 andcorresponding speech processing software, voice commands can likewise beprocessed using any of the many known speech processing systems. Susancan select a context, such as by spoken instructions, to be display. Asan example, Susan may request all photos related to “college”, relatedto “vacations”, including “family”, or including or relating to aparticular person. The digital picture frame will download relevantphotos for display from one or more electronic devices and/or socialmedia accounts, of Susan's and/or a community member of Susan, that havecontext matching the request. If Susan asks for pictures of her and hersister, the pictures may include her sister or have been taken in thepresence of both her and her sister, such as a picture of the EiffelTower taken during a trip with her sister to France. The Eiffel Towerphoto is identified by context metadata that identifies that both Susanand her sister were present when the photo was taken. The contextinformation is desirably automatically learned and stored with thepicture in the social media account of Susan, using the learning methodsdescribed herein. Alternatively, the Eiffel Tower photo can be part ofthe sister's social media account, and because her sister is a communitymember and has given sharing rights, the social media server computerautomatically shares the photo to Susan's digital picture frame.Desirably, any photo relevant to Susan can be automatically shared fromany of Susan's community members and automatically shown on the pictureframe display.

In one embodiment of this invention, the digital picture frame accessessocial media or her electronic device without detailed instructions fromSusan and loads and sequences photos displayed on the digital display asa function of profiling traits selected from chronological order, photolocation, photo activity, and/or community member. Having automaticallylearned the location, activity, and community member involvement on aper photo or video basis using the method described herein, a “storytelling” capability is supported. That is, in storytelling,chronological stories, optionally simultaneously displayed on a splitscreen, are grouped by: purely time, namely in sequential chronologicalordering; location, namely a traversal of sites on a location basedtrip; activity, namely in chronological ordering of a given or similarset of activities; community member involvement, namely a pictorialinteraction with community members, potentially segmented by particularcommunity member or members; or any other profiling trait(s) of arecognized user that can be used to cluster or segment photos forautomatic story telling.

In one embodiment of this invention, Susan's friend Mary joins Susan inviewing the picture frame, either directly or within a detectablevicinity of the picture frame. The picture frame automatically detectsthe additional presence of Mary, automatically detecting her identity,automatically obtains or loads Mary's profile, and automatically changesthe photos displayed on the digital display to those relevant to bothSusan and Mary. The photos could be obtained from Susan's social mediaaccount(s) or personal picture capture or storage electronic device butpreferably, the photos are obtained from both Susan's and Mary's socialmedia accounts and/or local personal picture capture or storageelectronic devices, e.g., mobile phones 29 and 29′. Desirably the photosfrom each of the two viewers are shared activity photos; from activitiesshared by the two friends. The frame 20 and/or server computer 26automatically determines shared activity photos of the two persons as afunction of the learned context information automatically associatedwith the shared activity photos by the server computer. As more peopleare automatically detected, photos from more accounts or devices can beautomatically added, and the photos automatically organized by contextsuch as photo location, a photo activity, and the present communitymembers. Options can be added on the social media accounts to share ornot share photos with community members or their devices according tothis invention.

In another embodiment, Mary and Susan each simultaneously, or withinsome specified duration of time, look at their respective digitalpicture frame 20 and 20′, such as linked as shown in FIG. 1 or networkconnected when each frame in a different location (e.g., theirrespective residences). Remote frames can communicate, via a network,such as through server 26, and indicate to both Mary and Susan, each attheir respective frames 20 and 20′, their respective presence,potentially indicating when Mary and Susan are both viewing or viewed aphotograph from a preselected set of photographs within the specifiedduration of time. The duration of time can be defined in terms of aspecific time or as within a time interval commencing from when eitherMary or Susan viewed the given photograph from within the preselectedset. The frames 20 and 20′ can automatically or upon instructioncoordinate so that the two are viewing the same photos. Likewise thephotos can be augmented to included shared experiences, and be uploadedfrom both of Mary and Susan's linked devices 29 and 29′ and/orrespective social media account.

In embodiments of this invention, photographs from digital photocollections are provided to the frame in multiple ways including but notlimited to self-capture, transmission from remote photograph capturingdevices, from other frames, and through server transmission. The frameeither receives photographs taken by other devices or itself takes aphotograph, if so configured. The digital photo collection(s) from whichphotos can be obtained and sent to the frame can be associated with oneor more additional viewers and/or digital picture frames, and/or anyelectronic device of a community member of the viewer.

Independent of the means by which a photograph is obtained, at times,the main artifact within the photograph, such as, for example, the faceof a person in the picture, is not properly aligned. As such, whendisplayed on the frame, either the main artifact is misaligned, orpotentially, is removed all together. To mitigate potentialmisalignment, the frame and method of this invention can include orprovide an automated smart cropping feature for automatically croppingand aligning the photos to fit the digital display of the frame.

For proper viewing, the frame or the server driving the disclosed framecan automatically crop and/or realign images. Without requiring userinteraction, the server executes cropping software that realigns animage for proper display on the frame to ensure that the main artifactwithin the image is automatically scaled and realigned to a given aspectratio (4:3 or 3:4 for example). The artifact may not be a single thingor entity, and, for example, could be a group such as two faces in theforeground close to each other or a tight group of people in a familyshot. An exemplary non-limiting approach identifies the face areas andattempts to align a corresponding area at either center to top-33% ofthe image to be portrayed, depending on the size. Large faces can becentered vertically whereas smaller faces can be aligned in a top 33% ofthe display to show more of the person's body. Thus, when presented, theimage is esoterically properly displayed.

Embodiments of this invention preferably also include image filtering,either at the frame or server level. Filtering can include removingimages of faulty artifacts, such as, for example, low quality (blurry,etc.) Images. Artifacts of unlikely interest can also be filtered. Forexample, photographs of pets containing faces are typically of interest,however animal photographs without heads are typically not of interest,and can be removed. The curator of the photographs, such as the owner orviewer of the frame, can additionally impose other image or contentquality constraints. Content constraints can span people, places,artifacts, and actions; namely anything that can be recognizedautomatically by any image, photograph, or document processingtechniques known in the art.

Embodiments of this invention enable the owner or user of a frame tospecify additional constraints on the photographs displayed. Such as inthe case of the photograph curator, the frame owner too may designatecontent restrictions; for example, pornographic photographs can beprohibited to display. Such constraining rules are defined andincorporated into the filtering approach. Note that it is also withinthe scope of this invention that the constraining rules include but arenot limited to lighting characteristics, scene selections, number ofindividuals displayed, distance or horizon specifications, or any otherphotograph characterization features.

The invention further includes receiver (displayed) side based filteringwhere the filtering of content to display to be defined by not only bythe curator/owner of the photographs but also the owner/user of thedisplaying frame. In the case of multiple owners/users of a displayingframe, the identity of the current viewer can be used to select thefiltering limitations imposed. It is also within the scope of thisinvention that a default set of filtering limitations are imposed shouldthe viewer not be identified, multiple viewers be detected, or any otherpre-specified set of conditions. Constraints imposed on the quality ofthe photograph to be displayed can be implemented, but also constraintsthat filter based on the sender's (photograph capturer's) desires;namely the sender imposed restrictions on the photographs to bedisplayed on the receiving frame. The receiver, namely the frame'sowner/user, also has the ability to limit the photographs to displaybased on their image quality or on their content.

The frame's owner/user provides the sender access to the frame. Thisaccess, without any receiver side filtering, aside from image qualityconstraints, gives the sender control over the content to be sent andcorrespondingly displayed on the receiving frame. By allowing thereceiver to limit those photographs that are to be displayed, a greatersense of comfort and security are provided to the frame's owner; hence,the frame's owner is more likely to allow additional senders to accessthe frame, and thus, the community associated with the given frame islikely to grow. Receiver-side filtering rules can include, but are notlimited to: general content restrictions, such as but not limited torestricting the display of nude photographs or photographs where toomany people are included; individual specific restrictions, such as butnot limited to restricting any photographs that include a previousspouse; or restrictions relying on metadata constraints, such as, butnot limited to, restricting photographs taken prior to a certain date ortaken at a certain location.

FIG. 2 illustrates an embodiment of the invention includingreceiver-side photo filtering. In FIG. 2 , Susan is at home with herdigital picture frame 20. Susan's daughter Sally and Sally's husband Bobare in a remote location, where Bob takes a picture of Sally on Bob'smobile device 29. Bob is a community member with Susan (and both withSally), and Susan has set content constraints to allow pictures of Sallytaken by any community member to be displayed on her frame 20. Inembodiments of this invention, when Bob takes the picture of Sally, thepicture of Sally is automatically identified on device 29 and routedover a network, such as via social media server computer 26, to Susan'sframe 20 for display.

To route the picture to Susan, the content of the digital photo isautomatically identified, and any target electronic device, such asSusan's frame 20, that would receive the photo also needs to beidentified and matched to the digital photo. The identified content isautomatically compared to content constraints of Susan's frame 20 todetermine if the digital photo should be routed to Susan's frame 20.Likewise other community member devices 29′ can be similarly processedeach time a photo is taken thereon to determine which, if any, photomatches or meets Susan's content constraints and should be routed anddisplayed on Susan's frame 20. Thus Susan can automatically receive, forexample, photos of each family member automatically upon the photosbeing taken. Susan can control the content and quality of the photos, toexclude, as examples, photos taken by community members of or includingnon-community members, poorly ‘angled’ selfies, photos taken with orthrough particular apps (e.g., Snapchat®), fuzzy, grainy or poorly litphotos, or duplicate photos (e.g., photos taken within a predeterminedtimeframe of each other). Photos not meeting Susan's content constraintsare desirably not routed to Susan's frame 20′.

In embodiments of this invention, content identification and/or arouting decision is/are determined by a content restriction filter. Thecontent restriction filter can be implemented via software coding storedand executed on, referring to FIG. 2 , the device 29, server 26, and/orframe 20. In embodiments of this invention, the filter components can beimplemented in modules on more than one device. For example, a photoquality filter module can be implemented on the photo-taking device 29,to limit photos sent to the server 26. The server can then performcontent identification on photos received, and route each photo to thetarget frame for which the photo meets the content constraints. Photoprocessing, such as cropping, aligning, scaling, enhancement, andrepetition filtering can be performed on the frame, server, and/ormobile device, as needed.

The digital photo frame 20 can additionally detect via the cameraSusan's physical response, for example a body movement or facialexpression, to the routed/displayed photo, and an interest level gradebased on the physical response can be associated with the photo. Theinterest level grade can be used to determine whether the digital photois replayed, or a frequency at which it is replayed. The interest levelgrade can additionally be used to automatically update the contentconstraints, such as if Susan indicates a low interest to one or morephotos of Sally with her tongue out, or with Sally at a particularlocation (e.g., a bar). In embodiments of this invention, the frame candisplay a message or message indicator requesting the physical response,for example so the frame can learn what photos to display.

FIG. 2 further illustrates a second digital photo frame 20′ of Susan'sfriend Mary, remote from the frame 20. In an exemplary embodiment, Maryis a community member with Susan, Sally, and Bob, but Mary has notadded/allowed photos of Sally or Bob in the content constraints of frame20′. However, Mary loves Paris, and wishes to see all photos related toParis from her community members. If Bob's picture of Mary with mobiledevice 29 occurs in front of the Eiffel Tower, then the contentrestriction filter module(s) will allow the photo to route to Susan'sframe 20 because Sally is present, and will also route the digital phototo Mary's frame 20′ for including content related to Paris. In this way,community members can learn of activities of other community membersthat are of particular interest, without receiving every photo from theother community members.

A non-limiting embodiment of this invention consists of at least onepicture capture or storage electronic device 29 communicating directlyvia a network or wired or other wireless connection with the digitalpicture frame 20. To support such interaction, the picture capture orstorage device 29 connects via the network to the digital picture frame20 and downloads the digital picture frame interaction software to thedevice 29. Once installed, the application software scans all theavailable pictures on the device. The remaining process description isillustrated using a non-limiting example based on face selection. It islikewise, however, within the scope of this invention to select based onobjects other than faces including but not limited to scenes, materials,and activities. A second device 29′ (or more) can likewise link to theframe upon being in connection proximity.

Once selected, all photos containing faces are clustered by theapplication, either at the server or frame level. Individual photographscan then be automatically filtered based on features. Some of thesefeatures are quality related, e.g., red-eyes or blurred; some featuresare content related, e.g., excluding certain activities, people, orlocations. Those photographs that remain selected can be tagged withadditional metadata. Metadata includes all photograph generatedinformation, such as but not limited to location, time, or weather, aswell as derived data, such as but not limited to cluster identity.

The invention can use clustering to form grouped clusters according tophoto content. The clustering is desirably performed as a function of acommon detected content in the photos, such as things, places,activities, or combinations thereof. The clustering of photos can beused for determining a photo slideshow on the digital display of thephotos from a digital photo collection as a function of the clustering.The grouped cluster can be integrated to a slideshow as preferredimages, or used to provide requested slideshows, such as “show grandma.”

Embodiments of this invention include identity clustering, such as bylongitudinal facial clustering. Photographs are clustered not only inaccordance to an individual identity at a given time, namely at a givenage, but the disclosed longitudinal facial clustering clustersindividuals throughout their lifetime. Consider a collection of picturesthat span a prolonged period of time. In such a case, an individualnaturally ages, and creating a collage of that particular individualnecessitates accounting for their aging process. The disclosed approachuses photograph metadata, including but not limited to the date thephotograph was taken, to age the individuals to a common age using anyof the individual aging techniques known in the art. Once at a commonage, clustering is performed to group pictures of the same individual,

FIGS. 2 and 3 illustrate a clustering of photos of Queen Elizabeth. Inone cluster (“Young Cluster”) are photographs of the young Queen. Herage progression is represented in various sub-clusters forming a photochain to a second cluster (“Old Cluster”). Note however that thesephotographs are all of Queen Elizabeth, and as such, they are alleventually clustered together (as represented by the linking edges) toform a larger cluster. The sub-clusters can be linked together byfeatures that do not change as much over time, such as eye distance,head shape, nose shape, etc. Clusters according to this invention can beconstantly updated and adjusted, such as using new photos and/or othernew metadata.

An alternative or additional clustering technique uses statisticalmodels of metadata. Using metadata, such as, but not limited to,location identification or date and time, the likelihood of a givenperson being at a given location at a given time is computed. Shouldphotographs support such a given likelihood, they too are potentiallyclustered.

Similarly, metadata can be used to prevent the clustering of an unlikelyset of photographs. For a non-limiting example, image-processingtechniques are somewhat inaccurate at recognizing and disambiguatingchildren. This can be particularly true for infants and young children.Photo metadata that indicate when and where a photograph was taken canbe used to reduce or eliminate disambiguation. For example, photographsof children taken at a close time interval but at vastly separatedgeographical locations are likely to be different individuals.Similarly, children of a similar age but appearing in photographs takenfar apart in time are again likely to be different individuals.

Clustering in additional embodiments of this invention employs a logicaltwo-phase approach. It is within the scope of this invention that the“two phases” are integrated into a single phase, or are partitioned overa greater number of phases, Initially, a cluster is formed using facialrecognition using any one or more of the many known in the artclustering techniques. This phase clusters photographs of individualsthat closely resemble across all photographs. However, should anindividual modify their appearance either intentionally, say but notlimited to using a costume, or unintentionally, say but not limited to alopsided profile, an erroneous separate cluster may result. Independentof cause, the merging of clusters of the same individual is needed. Thesecond phase uses known image processing techniques to extract featuresof photographs within the given cluster. These features are used torepresent the cluster, say via a representative centroid. By averagingthe features across the photographs within the cluster, the effects ofminor variances in some of the features are diminished. The centroids ofall clusters are compared. If centroids of differing clusters aresufficiently close, the photographs within the close clusters arefurther examined. This further examination includes but is not limitedto the use of respective metadata. If still deemed similar, the clustersare merged.

In another embodiment, the disclosed photograph clustering approachcombines clustering with classification. Current clusters form the basisfor inducing classification models. These models, in turn, are used toassign currently unassigned photographs to existing clusters if thesimilarity measure score employed meets or exceeds the discriminativethreshold. It is within the scope of this invention to employ any of themany known in the art image similarity measures.

In another embodiment, the linking of similar photographs across framesor digital photo collections is performed. Frame and/or mobile devicephotograph collections are maintained separately, however the submissionof pictures to a frame can constitute a community membership. Withinsuch a community, photographs of individuals portrayed might be common.In some embodiments of this invention clusters of photographs acrossframes/collections are compared and the server suggests to the owners ofthe frames, as well as to those submitting pictures to the frames, tosend photographs of common individuals to frames that already possessphotographs of those individuals, but not necessarily the identifiedphotographs. Thus, photograph sharing is fostered. It is also within thescope of this invention to impose a threshold on the minimum number ofphotographs that a particular individual must appear in a remote frameor other digital photo collection prior to suggesting that additionalpictures of that individual be routed to that remote frame.

Another embodiment clusters photographs based on common detectedobjects. Exemplary objects include artifacts (things, places,activities) and living beings (typically people and pets). Locations andactivities can be detected via metadata and individuals present can bedetected via facial recognition.

Photo data can be preprocessed on the photograph-capturing device. Theclustering and cropping approaches described herein assume theavailability of the data on a server. However, given current computingcapabilities on the photograph capturing devices, such as but notlimited to GPUs, significant preprocessing can immediately occur on theactual device. In doing so, the bandwidth required to transfer thephotographs to the server is vastly reduced. In embodiments of thisinvention, preprocessing includes, but is not limited to: image qualitydetection and enhancement (if photographs remain of poor quality thenexclude); content filtering (for example, pornographic or choppedfeature images are excluded); and repetitive (same picture takenmultiple times, only highest of quality is sent).

Additionally, picture fingerprints (compressed or very low resolutionversions) can be generated on the capturing device, and only thefingerprints are initially sent to the server. If the server deems thesepictures as desired, based on any pre-established criteria, the serverinforms the device, and the original pictures are transferred. Usingsuch an approach, only desired pictures are sent to the server, againreducing the bandwidth involved in the transfer.

Cross capturing-device filtering can also be incorporated. Using picturefingerprints, devices can interact directly among themselves todetermine which photographs to transfer to the server. For example, aset of community members may wish to strictly share images of commoninterest, and in embodiments of this invention the devices can sharefingerprints and automatically filter photographs of similar nature.Thus, only one of the photographs, potentially evaluated to be of thehighest of quality among the similar pictures, is transferred to theserver, again reducing the transfer bandwidth. Additionally, a votingprocedure can be imposed where only those picture with a sufficientlyhigh vote are transferred.

Likewise within the scope of this invention are other criteria by whichto select the photograph to transfer. Non-limiting additional oralternative criteria include but are not limited to the poweravailability or bandwidth capacity of the capturing device. As anexample, if a device has low bandwidth capacity or is nearly out ofpower, another device might be selected to send its version of a similarphotograph.

Photographs are posted to frames for the pleasure of the receiver,namely a viewer of the frame. In embodiments of this invention, theframe provides the frame viewer with the capability to provide feedbackto the sender of the photograph. Via a physical response, such as a handgesture, the viewer can send, if they so wish, an indication of thedegree of excitement or dissatisfaction with a given photograph. Oncethe gesture is captured and interpreted, the frame sends the server amessage indicating the viewer's interest level in the photograph, whichin turn, is stored and applied at the server level and/or indicated tothe sender of the photograph. Such a feedback loop motivates the sendingdevice or collection to automatically send or not to send similaradditional photographs. Likewise using facial recognition, the frame canidentify the viewer, and if desired, tag the feedback with the viewer'sidentity. Also the frame can infer feedback expressed via facialexpressions using facial recognition. Similarly, other body movementgestures, such as but not limited to the nodding or shaking of headrepresenting positive or negative feedback, respectively, can becaptured and treated as feedback.

The invention further includes ranking the photos to be circulated onthe frame, such as for the purpose of automatically determining an orderof presentation, a number of display repetitions, and/or a time periodof display for each of the photos as a function of the ranking. For anygiven frame, the photographs sent to the frame are circulated. Theordering of presentation of photographs for a given frame can be basedon a ranking. In one ranking, similar photographs are clustered basedon, for example, but not limited to people, places, things, oractivities. Photographs within these clusters are then shown insuccession or within a predefined, or viewer set, separation of eachother. In another ranking application, feedback provided by the frameviewer(s) is used to determine the ordering, with bias given towards themore favorable photographs. In yet another ranking application,photographs are presented in a random order. In yet another rankingapplication, popular photographs, based on feedback, are repeated,possibly even frequently. In yet another ranking application, allphotographs are shown prior to any repetition. In yet another rankingapplication, the time stamp of the photograph is alternatively oradditionally used for ranking. That is, a chronologically ascending ordescending order of photographs can be presented. In yet another rankingapplication, photographs are displayed according to similar seasons ordates within successive or separated years. In yet another rankingapplication, a popular photograph is displayed longer prior to beingreplaced by its successor. In yet another ranking application, theduration of display is the same for all photographs. In yet anotherranking application, photographs are clustered according to time slices,the number of photographs within each time slice is computed, andphotographs within popular time slices are shown more frequently. It iswithin the scope of this invention to vary the time slicessignificantly, whereby brief time slices with high photograph countstend to indicate great interest in the shown event, location, orindividuals. Lengthy time slices, without loss of generality, canaccount for trips, seasons, or any sustained activity or event. In yetanother ranking application, the viewer explicitly specifies theordering and duration of display. One skilled in the art recognizes thatany ranking rule known in the art can be imposed and that thesepresented rankings serve only as non-limiting examples. More so,multiple combinations of these and other ranking schemes are likewisesupported by this invention.

The digital frame photo systems of this invention can include and/orprocess great numbers of user photos. In such a massive photorepository, data can be extracted and/or trends can be detected and/oranalyzed. The invention further includes methods and systems for trend(or similar information) detection in the photo collections. Tofacilitate understanding, exemplary trends that can be detected andanalyzed from the processed photos include, without limitation,fashion/clothing, color schemes, locations, activities, etc. that arecurrently popular or growing in popularity. This information can befurther analyzed by location (which fashions, where), timing (e.g.,seasonal fashion or trips), the age of the individuals (if present inthe photos), occasion setting classification (e.g., socialgathering—formal or informal), etc. This information can be useful tovarious industries for design, production, shipping, forecasting,entertainment, etc.

The invention includes a method and system that extracts one or more ofcontent features from photos of the digital photo collection, and/orphoto image features from (same or different) photos of the digitalphoto collection. For example, deep learning systems can be used toextract image content features. The extracted features can berepresented by weighted n-dimensional feature vectors, which in turn,can be clustered using any suitable clustering techniques, includingtreating the feature vectors as documents, both using serial andparallel system technology, for example, as specified in U.S. Pat. No.5,864,855, herein incorporated by reference. Additionally, inembodiments of this invention, deep learners can be trained to simplycluster all photos based on readily available pre-trained conventionalmodels or using previously served training data. Similarly, photographscan be indexed, classified, and grouped using other technologies, suchas those specified in U.S. Pat. No. 7,787,711, herein incorporated byreference. The photos analyzed can be some or all of the photosprocessed by the overall system, such as by a photo-sharing systemdescribed herein. The photos and/or the extracted information can bestored in a database for determining correlations within/across thephotos, or at least groups of the photos.

In embodiments of this invention, the photos of interest, potentiallyall or selected for some criterion or criteria, without loss ofgenerality, due to the activities, locations, people, season, outfitsworn, etc. shown, are tagged with metadata corresponding to the detectedinformation extracted from the photos. For example, without loss ofgenerality, corresponding metadata can be maintained as a vectorencoding the items of interests found in the photo. Additionally,entries in each or some of the vector elements can represent specificcharacteristics, such as type of and color or texture of a clothingelement.

As an example, FIG. 5 shows a photo of a man and young girl at thebeach. The photo can be tagged as labeled based upon the colors, shapes,light intensity, and/or color histogram of the photo. The photo includesa man wearing a pink short-sleeve shirt and blue shorts, and the photocan be tagged with “man,” “pink” or “pink shirt,” “blue” or “blueshorts,” “shorts” and/or “short-sleeve shirt.” Other possible tags thatcan be obtained from the photo or metadata of the photo (such aslocation and/or date stamp information) include “beach,” “warm weather,”“tropical location,” and/or “vacation” (due to palm trees or being takenduring a winter month). The detected water can additionally oralternatively be determined to be, and tagged with, “ocean” in view ofthe beach and palm tree. Likewise the palm tree can lead to a “tropics”tag, and the sun can lead to a “warm” or “hot” tag, particularly in viewof the other photo features like the clothing worn. The photo in FIG. 5can likewise be tagged with “2 people”, “man and child”, “adult andyouth”, “father and daughter”, or any other such tagging. Facialrecognition and age-determination can be used to further identifyinformation on the person in the photo. If a number of analyzed photosacross one or more photo collections including photos from numerousdifferent users show pink clothing, or at least an uptick in pinkclothing in a location or timeframe, then a trend towards pink clothingcan be determined, such as for the location, timeframe or thecorresponding weather. As a further example, if spring break photos for2022 show increased pink shirts and swimwear, this information can beuseful to know for 2022 summer clothing trend forecasting, even byparticular genders or age groups.

It is within the scope of this invention to limit the tagging options.It is also within the scope to conflate multiple tag options into fewerand/or of fixed vocabulary. In embodiments of this invention, the photoextraction and/or analysis is limited to look for predetermined desiredinformation or features. For example, the photo extraction or analysiscan be limited to a particular clothing item/color trend (e.g., pink)and/or location trend (e.g., tropics or spring break) that may bedesired by a particular entity.

In embodiments of this invention the information detected in photos canbe clustered. The metadata clustering process for trend determinationcan be similar to that discussed above for frame display, however, isgenerally for a different purpose and using different criteria/photoelements, and can be used in addition to the clustering described hereinfor frame display purposes. As an example, tagged photos can beorganized into a plurality of sub-clusters, each for a correspondingcommon detected extracted content feature and/or extracted photo imagefeature, such as by activity, location, temperature/weather/seasons,time, attributes of a person of the photos, light intensity of thephoto, or a color within the photos, etc., and various combinations ofthese categories. Correlations between photos can be mined, such as bycomparing sub-clusters of photos across more than one digital photocollection over a network. Example correlations include determiningpopular locations, clothing items, colors, and/or activities within thetagged photos, such as for a predetermined time period and/or apredetermined age group.

Data correlations can be performed by any suitable approach known in theart. As an example, the method of this invention can operate in twomodes: mining and/or analytical processing. When mining, such as but notlimited to using association rule mining, the system generates a set of“association rules,” namely a set of implications with a confidence andsupport for these rules. In an alternative, the system can analyze,namely investigate or probe, the tagged metadata to better understandpatterns, such as using Online Analytical Processing (OLAP)

Association rule mining is one of many suitable data correlationapproaches that are within the scope of this invention in which a datamining process derives rules that may govern associations and causalobjects between sets of items. In a given transaction with multipleitems, association rule mining tries to find the rules that govern howor why such items are often bought together. The output is a set ofassociation rules that are used to represent patterns of attributes thatare frequently associated together (i.e., frequent patterns).

A data warehouse is a system used to report and analyze data thatsupport decision making. Typically, data from multiple sources areextracted, transformed and loaded into the warehouse. Then, analyticsare performed using, for example, Online Analytical Processing Server(OLAP), which is based on the multidimensional data model. OLAP is acategory of software that allows the system to analyze information frommultiple datasets at the same time. OLAP technology enables analysts toextract and view business data from different points of view. There arevarious OLAP operations to group, aggregate and join data such as rollup, drill down, slice and dice, and pivot (rotate). Roll up is used toaggregate on a data cube; drill down is used to reverse the operation ofroll up; and pivot is used to rotate the data axes in view to provide analternative presentation of data.

FIG. 6 illustrates a representative OLAP cube for a multi-dimensionaldataset, with each sub-cube having metadata tags corresponding to theaxes. The term “cube” here refers to a multi-dimensional dataset, whichis also sometimes called a hypercube if the number of dimensions isgreater than three. Data as a cube with hierarchical dimensions helpwith analyzing, and are easier to visualize. This dataset includes threecategories, namely Location, Time and Items. The illustrated locationsinclude Locations A-D, which could be countries, cities, schools,restaurants, sports fields, etc. The illustrated Times are shown asquarters Q1-Q4, which could represent calendar days, seasons, etc. Othertimes could be hours or daytimes (morning, midday, evening). Items W-Zcan be any photo content and/or photo feature from the metadata tags,such as an activity content (e.g., soccer, golf, beach, party, dancing,etc.), one or more attributes of a person or animal of the photos (e.g.,age, gender, race, breed, etc.), a light intensity of the photos (e.g.,bright, dim, night, day, etc.), and/or a color (e.g., of clothing,jewelry, vehicles, etc.) within the photos.

In embodiments of this invention, the correlation mining includesperforming slicing operations and/or dicing operations on themulti-dimensional dataset. Slicing and dicing refers to a way ofsegmenting, viewing and comprehending data in a database or datawarehouse. The term slicing and dicing is generally used in OLAPdatabases that present data in the multidimensional cube format. Toslice and dice is to break a body of information down into smaller partsor to examine it from different viewpoints for better understanding. Theslice is the act of picking a rectangular subset of a cube by choosing asingle value for one of its dimensions, and creating a new cube withfewer dimensions. The dice is the act of producing a subcube by allowingthe system to pick specific values of multiple dimensions. A maindifference between slice and dice in data warehousing is that the sliceis an operation that selects one specific dimension from a given datacube and provides a new subcube while the dice is an operation thatselects two or more dimensions from a given data cube and provides a newsubcube.

Slicing is illustrated in FIG. 7 , and dicing in FIG. 8 . Slicing is theact of picking a rectangular subset of a cube by choosing a single valuefor one of its dimensions, creating a new cube with one or more fewerdimensions. In FIG. 7 , the Location and Item of a particular quarterare “sliced” out of the data cube in FIG. 6 . The dice operationproduces a subcube by allowing the system to pick specific values ofmultiple dimensions. FIG. 8 shows a dicing operation, whereby the newsubcube includes Items (e.g., photo content) of a limited number for twoLocations and two Times.

Embodiment of this invention include, and the digital picture frame isimplemented with, a method, system, and/or apparatus, such as embodiedin an MPSM or other software application, that automatically determinesand shares a location, an activity, and/or photos of a user. Theapplication learns user activity over time, with the learning based uponuser locations and/or context. The application can learn throughautomatically determining activities at locations based upon knowncontext information and past context information for the location. Theapplication can tag photos for determining context relevancy for showingon the digital picture frame, as discussed above. The invention furtherincludes energy saving location methods for the mobile device that canbe used to more efficiently allow the location and social media aspectsof the invention to be implemented on a mobile device. The method andapplication can be used for any suitable function, such as a safetyand/or reminder serves, and is particularly useful for use in socialmedia applications and for generating photos for display on the digitalpicture frame. The invention will be described below with implementationin an MPSM system, and particularly with an MPSM application that learnsuser activity over time, with the learning based upon user locationsand/or context.

The MPSM method and system of this invention is mobile and positional innature. Such systems, like many other systems originally developed onone type of computing platform but migrated to another, operate not onlyon mobile environments. That is, while MPSM implementations are targetedto primarily execute on mobile devices, such as but not limited tosmart-phones, tablets, and/or laptops, they often support implementationfor non-mobile environments such as but not limited to desktops andworkstations, servers, and large scale compute farms and cloud computingservers. The invention will be described below with a mobile device,such as smart phone having cell service, a GPS system, and access to theInternet via WiFi.

The MPSM method and system of this invention is desirably executed orimplemented on and/or through a mobile device computing platform. Suchcomputing platforms generally include a processor, a recordable medium,an input/output (I/O) device, and a network interface capable ofconnecting either directly or indirectly to the Internet. The mobiledevice executes over a networked environment, a non-limiting exampleshown in FIG. 9 . The mobile device is connected, either directly orindirectly, using any of the many techniques and technologies known inthe art, over a network, to back-end system or systems,itself/themselves computing devices. The mobile device can connect witha remote server, shown in FIG. 9 as server 38, to store and/or accessuser or community information.

MPSM systems are used to support users remaining socially aware of theircommunity. That is, their primary usage typically is to actively monitorthe location and activity of family members, friends, colleagues, andgenerally others within one's community. Communities can be partitionedinto sub-communities where the union of the sub-communities forms theuser's community. The sub-communities may or may not overlap. Thepartitioning of communities into sub-communities is beneficial insupporting specialized applications. For example, while a user mighthave general interest in the location and activity of all of theircommunity members, they might be particularly interested in the locationand activity of those who might be suddenly in need of assistance.

The creation of a community can include the issuing of invitations. Aninvitation is a request by a user A of another user B to allow theinviting user, user A, to track the activities of the invited user, userB, and vice versa. If the invited user accepts, the inviting and invitedusers form a community.

A community is relevant to only that user which formed it. That is,different users have different communities. A community is a grouping ofinvited (referred to as remote) users by the inviting (referred to aslocal) user. A local user can partition or merge a community, thusforming a sub-community or a parent community, respectively. Forexample, consider 5 users: Bob, Sam, Sally, Alice, and Susan. Bob caninvite Sam, Sally, and Alice, thus forming his user community. Bob canlikewise partition his community into a sub-community consisting of onlySam and Sally. Sally can invite Susan. Thus, Sally's community wouldinclude Bob (via his invitation) as well as Susan. If no additionalinvites occurred, Sam's and Alice's respective communities would onlyinclude Bob (each via Bob's invitation), while Susan's community wouldonly include Sally (via Sally's invitation).

Providing users with the opportunity to expand their communities in aconvenient manner is advantageous. Such expansion can seamlessly beaccommodated by including users listed in a user's contact lists eitheras a whole or selectively into their community. Contact lists include,but are not limited to, users listed in a user's local address book,e-mail contact list, Twitter follow list, LinkedIn connections list,and/or Facebook friends list. By incorporating users listed in a user'scontact list, the user's community is expanded without effort. Note,however, that selected inclusion can be supported; thus enablingcommunity growth without unnecessarily over-expanding the community.That is, entries from the contact list can be included in their entiretyand the user can selectively remove those entries which s/he wishes tobe excluded from the community. Similarly, entries from the contact listcan be selectively added.

Users are identified by their account identifier. To use MPSM a useraccount is created. User accounts generally require a user login, whichis a unique user identifier, and a password or equivalent. After havingcreated an account, a user can log in. Initially, the local user doesnot have a community. In embodiments of this invention, over time, themethod and application tracks the activities and location of the localuser. Should the local user establish a community as aforementioneddescribed, the community members will likewise be tracked. Local usersreceive notifications of the location and activities of their communitymembers. Once logged in, the local user can select to activate ordeactivate self and community tracking and notification. If notoverwritten, default settings are used.

Whenever logged in and tracking is enabled, a user's location andactivity is tracked. That is, a user periodically records their locationand/or activity. Locations are tagged by name. Names can be but are notlimited to the following schemes: physical (e.g., 123 Oak St.), absolute(e.g., Acme Coffee), and/or relative (e.g., my work office), orproximity (e.g., two miles from home). Activities are typically events.These events might be common to the entire community such as: “drinkingcoffee,” “eating lunch,” “sampling wine,” “working from home,”“commuting,” etc., to more specific to a local user such as “restoringcar” or “driving to lake home.” Multiple activities can occursimultaneously. Users can change their activities at any time.

Unless preloaded or derived from an external source, such as but notlimited to a location database, initially, all locations and activitiesare unknown. Local users must record all such location-activitycombinations, i.e., a local user must name or tag the location and theassociated activity. A list of activities common to the local user'scommunity can be provided. This community activity list can be rankedeither arbitrarily (randomly), according to most recently used, mostfrequently used, relevance to location, alphabetically, etc. Eventually,an activity list specific to the local user is learned. This local useractivity list can be displayed to the local user either individually,along with the community list, or merged with the community list. Again,any of these lists can be ranked as previously mentioned.

FIG. 9 illustrates a representative area 30 to demonstrate a method ofand application for locations and/or activities of a user participatingin a social networking service. The area 30 is shown as a cellularcommunication network including a plurality of cells 32 each disposedaround a cellular communication antennae or base station 36. Within thearea are a plurality of destinations each shown as including a WiFiInternet connection. The local user has one or more electronic devices,such as a mobile device that communicates with a remote server 38 viathe cellular network and/or the WiFi connections. As will be appreciatedthe methods and applications of this invention can operate within anysuitable size and configuration of the communication area, depending onwhat the user encounters.

Destination 40 is the home of the user. The user commutes to office 42for work on most business days. On the way the user typically stops atthe coffee shop 41. For lunch on most days, the user visits restaurant43, but on Wednesdays the user typically meets a second user for lunchat restaurant 44.

At each destination 40-44, the user enters user information about thedestination. The application and computer system that receives the userinformation automatically associates the user information with thedestination, and stores the user information in a locations database,such as on the device and/or at server 38. The destination desirably isdetermined automatically and tagged with the user information, such as alocation name of the destination and/or the user activity beingperformed at the destination. For example, destination 40 can be taggedas “home” and likely has numerous activities associated with it. Thedestination 41, and any photos taken, can be tagged as its establishmentname “Acme Coffee” or simply “coffee shop” and associated with the useractivity of “buying coffee” or “latte time.” The manually entered userinformation can then be automatically shared to the user's community ina social networking service. Similar user information is received forthe other destinations 42-44. The user information desirably includesany other information about the location or activity, whether manuallyentered or automatically determined, such as the time of the visit oractivity. Some destinations, such as home or work will likely havemultiple user activities over a period of time, such as “coffee break,”“meeting time,” and/or “quitting time.”

The computer system receives user information and associates the userinformation with the corresponding destination, and any photos taken,for multiple visits to each of the destinations 40-44. The computersystem begins learning the locations and user activities. In embodimentsof this invention, the user can be automatically prompted forconfirmation of the user information upon arriving at a destination toconfirm the location and/or user activity. For example, the user can beprovided with an automatically generated list of previously entered useractivities for the destination upon arrival, thereby promoting efficientcollection of information. The items on the list can be listed in anorder based upon a particular ranking, such as number of times enteredpreviously or based upon a context, such as what activity is likelybeing performed at a particular time of a particular day.

Over time, the computer system learns the user information and beginsautomatically associating and identifying at least some user activitiesfor corresponding locations and any photos taken. As will beappreciated, the automatic identifying of activities at locations willlikely occur at different rates for different activities and locations,with some locations having fewer activities and/or more frequent visitsthan others. In preferred embodiments of this invention, the systemautomatically shares the user information in a social networking serviceupon automatically detecting further user arrivals at the destination.Photos taken are likewise automatically tagged with the userinformation. The automatic sharing of user locations and/or activitiesdesirably occurs upon the user's arrival at the location, or at aparticular time at the location. As such the invention includes anautomatic detection of the user's arrival at a destination. Theautomatic sharing and photo tagging desirably operates without useraction and prior to receiving any additional user information for thedestination.

As an example, the user may typically purchase lunch at destination 43,but on Wednesdays typically goes to lunch with a friend or spouse atdestination 44. The lunch routines of the user, and particularly theWednesday lunch routine, can be learned by the system and automaticallyshared to the user's community upon the system automatically determiningarrival, without manually input from the user. If the user is havinglunch with a community member, then the system can automaticallydetermine that both users are at the same location together toautomatically recognize and confirm the lunch activity, and proceed toautomatically share the information for both user's to their respectivecommunities. If the user deviates from a routine, the system can knowthis, and refrain from sharing the typical destination, by the mobiledevice detecting a different location than the typical routinedestination.

In embodiments of this invention, learning is accomplished by any knownmachine learning, data mining, and/or statistical techniques known inthe art. Supervised, semi-supervised, and/or unsupervised approaches canbe used, including, but not limited to Naïve Bayes, Neural Networks,Support Vector Machine, and/or Associating Mining based techniques.

The MPSM method and system of this invention desirably records allposted locations and activities. Throughout use, the disclosed inventionlearns the corresponding locations and the set of associated activities.More so, via comments made by the local user and by the local user'scommunities, the importance of the activities can be learned, such asfor the prompting discussed above. Importance can be either local useror community biased. Additionally, importance can be biased by context.For example, community members as a whole might prefer “eating steak,”“eating pizza,” and “eating sushi,” in that respective order. On theother hand, a local user might only eat sushi. Thus, local user biaswill yield “eating sushi” only, while community bias will suggest“eating steak,” “eating pizza,” and “eating sushi,” in that respectiveorder.

In embodiments of the MPSM method and system of this invention,locations are named according to a naming convention. Regardless of thenaming convention used, a location is a physical geographical position.More so, physical geographic locations associate properties that canvary with or be dependent on context, namely time and date (hours, dayof week, calendar date, etc.), users involved, and their relationshipsto each other, etc. This context can affect the associated location nameor activity.

A common scheme that can be used to at least assist in identifying aphysical geographical location is via the use of geocoding. Geocoding isthe representation of a physical location via the pairing of latitudinaland longitudinal coordinates commonly referred to as a lat-long pair.Global Positioning Systems (GPS) can also determine a physical positioncoordinated via the triangulation of satellite transmissions. TypicallyGPS devices derive lat-long pairs which are made available to a varietyof applications, often via map displays. GPS economics, accuracy, andsimplicity of use resulted in their wide appeal and commercial success.Their continuous use in mobile devices is problematic, however, as theyare energy intensive and rapidly drain the battery. Thus, alternativemeans or approaches to detect locations are desired.

Embodiments of the MPSM method and system of this invention, asdiscussed above in FIG. 9 , use or rely upon cell coordinates. Whenmobile devices communicate with a cell tower, they send their cellcoordinates. These coordinates are recorded by the cell provider and aretypically not publicly known. The cell phone or, in this case, themobile device supporting the positional social media system, however, isaware of their coordinates. Thus, the device can store the cellcoordinate position and automatically associate that cell coordinatewith the location name provided by the local user. Over time, a locationdatabase of cell coordinate and named location pairs is created. Thelocal portion of the database favors the local user. The union of allthe local portions of the location database desirably constitutes thename space of the entire MPSM system of this invention. It is understoodthat any of the many database management systems or storage schemesknown in the art can serve as the platform for this location database.Thus, location names can be provided without the need to rely on aglobal positioning system, reducing battery consumption. Location datacan additionally or alternatively be purchased or otherwise provided bya third party.

An additional and/or alternative approach for automatic locationdetermination relies on WiFi triangulations. Mobile devices can grow andmaintain a database of known open WiFi networks, for clarity we callthis database an Open-WiFi-Net database. Such mobile devices can use theinformation stored or derived from the information stored in theOpen-WiFi-Net database to further refine the accuracy of a locationwithout the use of GPS. Via point triangulation, when an Open-WiFi-Netdatabase is available, the mobile operating system uses not only thecell tower but also WiFi triangulations to determine location. It iswithin the scope of this invention to use either or both cell-phone andWiFi triangulations to enhance location information in addition to anyother disclosed approach. The mobile device can use the WiFi signal at adestination, such as destination 43, and additionally or alternativelyany detectable open WiFi signal from a neighboring location, such asestablishment 45 that is adjacent destination 43.

Having created the location database, searching, namely querying, thedatabase uses the cell coordinate or the location name. That is, alocation name query takes a location name as input and returns thecorresponding cell coordinate. A cell coordinate query takes a locationname as input and returns the corresponding location name. Note that,multiple names can be attributed to a given cell coordinate. That is, alocal user might name a location using multiple different names;different users can name same locations using different names.Similarly, the same name can be used for different cell coordinatelocations. All names corresponding to a given cell coordinate arereturned. It is within the scope of this invention to selectively returnnames based on context, user, or community bias. Similarly, all cellcoordinates corresponding to a given name are returned. Again, it iswithin the scope of this invention to selectively return coordinatesbased on context, user, or community bias. Ranking of the resultsreturned can, when desired, be biased towards the local user.

A key concern for MPSM systems is collecting location information.Clearly any location information available within the mobile deviceshould be harnessed. Thus, if GPS readings or any other locationinformation is generated by other device resident applications, thesereadings are desirably recorded and utilized by the method andapplication of this invention. However, reliance on strictly otherapplications to obtain positional information is obviously not realisticor possible.

In embodiments of the MPSM method and system of this invention,positional information is obtained via the use of geofences. A geofenceis geographical boundary or “fence” surrounding a positional reading. Asthese boundaries are radius based, geofences are generally circular.Location transmission occurs whenever a handover of one cell tower toanother occurs and is expected but not guaranteed to occur once ageofence boundary is crossed. To track location, periodic locationtransmissions are required. Since location transmissions must beminimized to conserve device energy, transmissions should only occurgiven geographical movement. Thus, crossing a geofence should generatesuch a transmission. Unfortunately, as crossing a geofence does notguarantee a location transmission, increasing the likelihood of atransmission is necessary.

In contrast to the known uses that surround a location with a singlegeofence, to increase the likelihood of a location transmission duringmovement, embodiments of this invention include surrounding a locationgeofence with a plurality of geofences. In one embodiment of thisinvention, a method and system of tracking a user includes determining alocation of the mobile user, automatically establishing a first geofencearound the location, and automatically establishing a plurality ofadditional geofences around the first geofence, with each geofenceincluding a boundary. A location transmission is obtained by the mobiledevice upon crossing a boundary of the first geofence or any of theplurality of additional geofences. Multiple neighboring geofences areadvantageous since they increase the likelihood of a locationtransmission as their boundaries are likewise likely to be crossed givenmovement.

FIG. 10 representatively illustrates a geofence 60 surrounding a currentlocation 62. The geofence 60 is surrounded by additional geofences 64,all within a given cellular tower transmissions cell 65. Note that partof a neighboring geofence 64′ is not fully within the cell and hence,limits its benefits since a cell tower handoff by movement into cell 65′will generate a location transmission.

Geofences are implemented as software processes. Operating systems formobile devices, such as but not limited to iOS and Android, limit thenumber of processes available to an application, and thus, the number ofgeofences is bounded. However, this limit typically exceeds the numberof geofences generated using the approach described above. Therefore,additional processes are available, and hence, additional geofences arepossible.

To increase the likelihood of a location transmission given movement, inembodiments of the invention, the remaining available processesimplement static geofences. A static geofence is not dynamicallygenerated given a new location as previously described. Rather, a staticgeofence is one that is fixed and represents those locations that arelikely to be crossed by a given user. That is, users are habitual andfrequent a limited set of locations often, for example but not limitedto, their home, office, or favorite wine or sushi bar. By learning thefrequent locations of users both individually and system wide andsetting static geofences at these locations, biasing by the individualuser, the probability of a location transmission is increased sinceadditional geofences are likely crossed.

More so, these repeated locations vary by city, county, state, country,etc., as well as by other factors such as but not limited to day andtime. Geographical and temporal presence can thus be used to vary theset of static geofences for a given user. For example, the set of staticgeofences for a given user will vary if the user is in Washington, DCrather than in San Francisco, CA. Similarly, the set of static geofencesfor a given user will vary depending on the day and time. For example, auser frequents work on weekday mornings but frequents their favoritebagel shop on Sunday mornings and their favorite sushi bar on Thursdayevenings.

Location transmissions suffer from a margin of error. Thus, it isdifficult to precisely pinpoint and tag a location with a singletransmission. Embodiments of this invention include automatic refiningof a location of a user destination as a function of user routines, suchas established by several user visits to the destination. As timeprogresses however, and a user frequents the same location multipletimes, multiple location transmissions for the same location arerecorded. In one embodiment of this invention, as representatively shownin FIG. 11 , by overlapping the transmitted location along with itsmargin of error, a more accurate location can be derived. Theoverlapping of location transmissions for a given location 70 betweenstreets 72 and within geofence 74, along with their margin of errors,represented as circles 76, yields an accurate location placement.

As shown in FIG. 11 , location accuracy improves as related data arecollected. Related data, however, can, at times, be somewhat erroneous(in terms of accuracy). A non-limiting example is an entrance to ashopping mall. Such an entrance is not necessarily at the center of thecomplex. Regardless of the entrance displacement from the center of thecomplex, the entrance location can still be used to increase locationaccuracy of the mall complex since the readings for the entrance areconsistent. That is, for a given user, given mobile device, givencarrier, etc., such location recordings remain consistent, all be it,slightly erroneous. Thus, even dirty, namely potentially inaccurate,data can result in correct location identification.

Additionally, having established a location, corresponding lat-long paircoordinates can be reversed engineered, namely mapped back onto, a placename. These derived lat-long pair coordinates become yet an additionalinformation component that is used by a learning system to better refinea mapping to a named place. Machine learning, data mining, andstatistical approaches that are supervised, semi-supervised, orunsupervised can be used, as known in the art, to cross-correlate allavailable location related data.

Once determined, the user information including the location and/or theuser activities are automatically stored in a database. Embodiments ofthe MPSM method and system of this invention include a computer serverfor providing and implementing the tracking and/or social networkingservice of this invention. The computer server includes a locationmodule to determine the user location and/or a tagging module configuredto correlate manually entered user information to a user destination anda database module configured to store user information including userlocations and user activities at the user locations. For social mediaand photo sharing, the server further desirably includes a communicationmodule configured to automatically share a user activity or photo in thesocial networking service upon further user arrivals at a correspondingone of the user or community locations. The server can also include anassociation module configured to associate the user activity with thecorresponding user location and any photo taken.

Since location transmissions are needed during movement, the obviousquestion arises: when should the transmissions cease? That is, thesystem must determine when the user has arrived at a location to knowwhen to perform the automatic steps discussed above. As discussed above,GPS systems are an energy drain on a mobile device, particularly as theGPS remains on and linked with the satellites to maintain locationdetection. Keeping a GPS application operating is a drain on both theprocessor and the battery of the mobile device. This invention providesa method and executable application that conserves energy by notcontinually running during use of the mobile device.

Embodiments of the MPSM method of this invention provide an automatedmethod of tracking a mobile user that includes providing a locationmodule configured to receive location transmissions, placing thelocation module into a sleep mode, awakening the location module uponreceipt of a location transmission, and determining a location with thelocation module. These placing, awakening, and determining steps arerepeated, thereby placing the application into a sleep mode when notneeded, thereby reducing the drain on the mobile device. The applicationgoes into sleep mode when necessary or when desired, such as when theapplication is not needed, e.g., during extended movement or upon anarrival at a location. In embodiments of the MPSM method and system ofthis invention, the application can go into sleep mode whenever a timesince the device awakening exceeds a predetermined time allocation, orupon a determined rate of travel exceeding a predetermined threshold,thereby indicating extended travel.

FIG. 12 illustrates one exemplary, and non-limiting, method according toan embodiment of this invention to automatically detect arrival at adestination. The method is useful for tracking a user's location for anyof various reasons, including, for example, for safety, to provideautomated reminders, and/or to provide automated suggestions to the userbased upon the destination and/or surrounding area. The method of FIG.12 is particularly useful for implementing the method and systemdiscussed above, and can be used to implement other applications andmethod to provide energy savings compared to GPS location methods inmobile devices.

FIG. 12 includes a flow chart 100 that includes and/or represents threedistinct situations, namely, an actual arrival, rapid movement, andsporadic movement without an actual arrival. Initially, the applicationis in sleep mode. Sleep mode is a state when no processing, and hence noenergy consumption, takes place. Processing occurs once the applicationis awoken. A location transmission, such as a cell tower transmission oranother application obtaining location information, awakens theapplication in step 102. Since the application awakening occurs due to alocation transmission, the current location is known.

Once awakened, the application typically has a maximum amount of time tocomplete its processing. This limit, called time allotment, is set bythe device operating system. All processing must complete prior toexceeding the time allotment. Ideally, the application should relinquishthe processing flag back to the device operating system before theoperating system forcefully removes the application from its activequeue. Voluntarily terminating an application, namely returning it tothe sleep mode, rather than having it forcefully terminated by the hostoperating system, is consider good citizenship. In step 104, theapplication initializes two timers, namely, a timer count representingthe duration of time the process has executed since last awakening, anda stationary count representing the duration of time since the lastdetected device movement.

As time progresses and the process executes, the timer count isincremented in step 106. In one embodiment of this invention, wheneverthe application processing time exceeds the operating system timeallocation (108—YES branch), the application is voluntarily placed insleep mode 105. Note that the time allocation threshold is notnecessary, but set to support good citizenship.

Assuming that the time limit has not been reached (108—NO branch), theapplication waits for t time units in step 110. After waiting t timeunits, new current location data are obtained is step 112 and storedlocally on the device in step 114. In step 116, the current location iscompared to the previously known location. If the two locations differ(116—NO branch), the rate of travel is computed in 118. If the rate oftravel exceeds a threshold (120—YES branch), the process is desirablyand voluntarily placed in sleep mode 122. Rapid travel is unlikely toresult in an immediate or near term arrival; thus, checking locationswhile moving rapidly unnecessarily uses device energy. Eventually, theapplication process is awoken with the device moving at a slower rate.At that time, location checking is needed as an arrival might soonoccur. If or when the rate of travel is slow (120—NO branch), movementis noted in step 124, and the loop is repeated commencing with theindication that additional processing time has elapsed in step 106.

Thus far, the arrival detection process has been voluntarily placed insleep mode either due to having exceeded the self-imposed processingallotment quota which is desirably set just slightly below the operatingsystem's time limit that leads to the removal of the application fromthe active queue (108—YES branch) or having travelled too rapidly(120—YES branch). Slow travel has resulted in simply recording thelocations traveled, noting the movement exists in step 124, and awaitingeither arrival or process termination.

Arrival is determined when the same location is detected for asufficient duration of time. That is, an arrival is potentiallydetermined when the location remains the same (116—YES branch). Thestationary detection count is then incremented in step 126. If thestationary threshold is not yet exceeded (128—NO branch), theapplication waits for t time units in step 110, and the current locationis obtained in step 112 and stored locally in step 114. A sufficient andpredetermined duration at the same location eventually surpass thearrival detection threshold (128—YES Branch).

Once arrival is determined, arrival is declared in step 130, all dataregarding the prior locations visited and stored locally are compressedand sent to the back end system supporting the application in step 132.A new location checkpoint is established in step 134, and the process isplaced in sleep mode 136. From the sleep modes, the process of FIG. 12repeats upon a known location.

Compression of location data is typically performed prior to localdevice to back-end system transmission as often the location data maynot be needed at the back end. Location data may not be needed in cases,for example but not limited, during rapid travel. Although exemplifiedas having data compression occur prior to the sending of the data to theback-end, it is within the scope of this invention to compress locationdata prior to storing them locally.

All parameters described above for FIG. 12 , for example t (for the timeunits), timer count, etc., are system and device dependent.Experimentation with and fine tuning of these and other parameters iswithin the scope of this invention. Also within the scope of thisinvention is the tuning of these and other parameters via the use ofmachine learning, data mining, and statistical approaches; supervised,semi-supervised, and unsupervised approaches can be used.

As discussed above, once the user has arrived at a destination, thelocation identification, user activities at the location, and/or anyproximate third party members of the user's community are determined, ifnot already known. In this way, the devices automatically continuallydetermine locations which can be used to identify any establishmentsand/or any community members at or within proximity to the location.

User activities are actions or events. Example user activities includebut are not limited to “drinking wine,” “flying,” “reading,” “attendingconference,” or “commuting.” Users specify a particular user activityeither by selecting from a provided list or by entering a different useractivity. As discussed above, the provided list is generated by storingall previously entered user activities from all systems users butbiasing the ranking of the provided activities based on context, thelocal user, their community, or a combination thereof.

All location and user activity pairs are stored in a databasecorrelating the location with the activity. Any of the many databasemanagement systems or storage schemes known in the art can serve as theplatform for this location-activity database. Furthermore, it is wellunderstood in the art that the location-activity database can store manyadditional features. For example, the user identity and date and time ofthe pair are recorded.

Over time, the database grows and contains a sufficient number of pairsto support mining. The volume of data needed to mine correlations isdependent on the mining algorithm deployed and the level of accuracyneeded. As known in the art, there are many machine learning, datamining, and statistical approaches to support mining. By using any ofthe many available such approaches, either individually or incombination, a local user activity preference per location is learned.Example learning approaches include supervised, semi-supervised, andunsupervised approaches including but not limited to Naïve Bayes, NeuralNetworks, Support Vector Machine, and Associating Mining basedtechniques. The use of proprietary mining techniques is likewise withinthe scope of this invention. Once local user preference is learned, thispreference is used to bias the aforementioned provided user activitylist.

There are many approaches to identify locations. Automated locationidentification is accomplished by periodic checking of the currentlocation. Periodicity of checking depends on, for example, the methodused to determine the location, the desired frequency of reporting,recording, and notification, and the resources available to support thechecking. Other periodicity determination approaches known in the artcan likewise be used. One approach to automate location identificationis the periodic determination of lat-long pairs via the use of a GPSdevice. An online service or a locally resident database can be used tocorrelate the GPS readings with locations. A preferred embodiment ofthis invention uses the aforementioned location database. Whenever atransmission to a connected cell tower is made, the cell coordinates ofthe transmitting device are used as a search query against the locationdatabase. If a match is detected, that location is identified. Anotherpreferred embodiment detects locations upon the crossing of geofenceboundaries as previously discussed. Note that both dynamicallydetermined geofence boundaries and static geofence boundaries detect alocation. Yet another preferred embodiment detects locations bycapitalizing on location transmissions generated by any otherapplication operating on the mobile device requesting locationinformation.

In embodiments of the MP SM method and system of this invention, localusers, unless disabled by a local user, can be provided with automatednotifications for themselves and for their community members. Thesenotifications describe locations, activities, or correlated locationsand activities for themselves and their community members. For example,unless disabled by the user, any time a user arrives at a new location,the local user and their communities can be notified of the user's newlocation. Automated location detection and notification, unlessdisabled, occurs without requiring a local user prompt.

Similarly, activity notification can be automated. Once a user arrivesat a location, a set of activities previously occurring at that locationis shared with the community or provided to the local user forinformation or sharing. If the user chooses to confirm at least one ofthese past activities, both the local user and their respectivecommunity members are notified of this at least one activity, and anyphoto taken is automatically tagged with the context information.

In another embodiment of this invention, automated notification involvesshared experiences. A shared experience is one that associates multipleusers. These associations can be passive or active. A passiveassociation is strictly informative in nature while an activeassociation requests an action. Non-limiting examples of passive sharedexperiences based on locations include: “User A is at User A's office,as is User B” and “User A is at home as is User C.” Note that the firstexample involves multiple users at the same physical location, namelyUser A's office, while the second example involves multiple users at thesame relative locations, namely their homes, but at different physicallocations.

Similarly, passive shared experience notifications can be based on useractivity. Non-limiting examples of passive shared experiences based onactivity include: “User A is eating lunch as is User B” and “User A isparticipating in her favorite sport as is User B.” Note that the firstexample involves multiple users participating in the same activity,namely eating lunch, while the second example involves multiple usersinvolved in similar nature of activities, namely participating in theirown favorite sport, which can be different actual activities, namelyracquetball and swimming. In both passive shared experiences based onlocation and on activity, known in the art machine learning, datamining, and statistical approaches that are supervised, semi-supervised,or unsupervised approaches can be used to correlate relative locationsand activities to physical locations and activities.

Other shared experiences can prompt for action, and are thus consideredactive. A non-limiting example of an active shared experience promptingfor action includes: “User A posted a picture when at Penn Station; youare now at Penn Station; please post a better picture?” Thus, activeshared experiences request the user to actively react. As above, activeshared experiences can be location or activity based and can be absoluteor relative. Note that it is likewise within the scope of this inventionthat individual user notifications be active and passive, in a similarmanner as described above. However, the correlation of locations andactivities both for passive and active are based strictly on thecurrent, past, or projected expected activities of the individual userrather than those of multiple users.

Typically, only changed locations and activities are notified. That is,a location or activity is not typically repeatedly identified. However,a local user can request repetitive notifications based on anytriggering condition known in the art.

Local users do not always remember to indicate a new location name orconfirm which of the possible suggested name or names the systemindicated for the given the location. As such, it is at timesadvantageous to prompt the local user for information. However, overlyaggressive prompting might annoy the user. In embodiments of thisinvention, the application non-invasively prompts the user upondetecting an unknown location for the given local user. To avoidannoyance, prompting is repeated only rarely, say twice; the number ofrepeated prompts can be set as a parameter. Similarly, to provide asense of comfort, if the back-end system recognizes the location basedon the local user's community members' naming schemes, it prompts thelocal user with guiding messages, for example but not limited to “Manyof your community members call this location The Tasting Room.”

Identification of activities associated with a given location or a givencommunity member can be additionally or alternatively automaticallyinferred in multiple ways. In embodiments of this invention, thecomputer system can automatically determine a positional destination ofa user, such as by using a mobile device discussed herein, andautomatically deduce as user information a location type and/or useractivity of the positional destination. The user information can bededuced, at least in part, based upon the destination context. Exemplarycontext information includes, without limitation, time-dependentinformation (e.g., what time of day is it?), community information(e.g., who is also there?), and/or third-party information about thepositional destination. This method, tied with automatic sharing of theuser information in a social networking service, can provide a partiallyor fully automated process for determining user location and activity,and tagging photos taken with the context information.

In one embodiment of the MPSM method and system of this invention, theautomatic deducing of the user information is based upon known orlearned user routine. As discussed above, local users typically followstandard routines. Some routines are daily, weekly, monthly, etc. Otherroutines are context dependent. Regardless of the nature of the routine,learning via any of the many statistical, machine learning, data mining,or business analytical techniques known in the art, enables predictivebehavior and automated activity and location suggestion. For example,but not limited to, if a local user always goes out to lunch at noon onevery weekday, then if an unknown location is detected on a Tuesday atnoon, then the application can suggest that this unknown location islikely a restaurant and the activity is likely eating lunch. Similarly,routine identification enables the prevention of transmissions bothpositional and informational. For example, but not limited to, if alocal user always goes to sleep at midnight on Sunday through Thursdayand awakens at 7:00 am the following day, then energy can be saved ifthe application voluntarily places itself in sleep mode during the hoursthat the local user is known to be sleeping. Additionally, routines caninvolve a sequence of activities and locations. A non-limiting exampleof a sequence of activities includes: On weekdays, Eric arrives at hisoffice at 8:00 am, drinks coffee at 10:00, develops software from 11:00am until 5:00 pm, commutes home at 5:30, and finally, arrives at home at6:00 pm.

Another location and/or activity deduction approach is by association.The automated deducing can include automatically associating a user witha second user at a positional destination. If the second user's locationand/or second user's activity is known, then the system canautomatically infer the location type and/or user activity of the firstuser from the second user location and/or activity. Consider a previousknown event such as: “Community member Sally swimming at the Somersetpool,” assuming that the Somerset pool location was previouslyidentified. As an example of automatically determining a currentactivity of community user Sam, the system identifies through locationdetermination that Sam is currently at the same location as Sally, andalso that Sally is currently at the Somerset pool. From thisinformation, possible automatically postulated associations andactivities are: “Sam is at the Somerset pool,” “Sally is swimming,” and“Sam is swimming,” Thus, it is possible to infer an activity for acommunity member from association with another community member. It iswithin the scope of this invention to use any logical inference methodsknown in the art to generate plausible associations. It is also withinthe scope of this invention to obtain confirmation of the plausiblepostulated activity by the community member, in this case Sam, by askingeither Sam or Sally or by any other means known in the art.

Desirably the computer system operating the MPSM automatically storespast user information, including past location type and/or user activityof the positional destinations of all users. User information for futurevisits to repeat positional destinations can be automatically deduced asa function of the stored past location type and/or user activity of thepositional destination. In embodiments of this invention, the system canrely on recorded previous activities of a user, a community member, orany system user at a given location to postulate on a user's activity ata given location. Past context information for past visits to thepositional destination by the user and/or community members of the usercan be compared to a current context of the user's visit to thepositional destination to deduce the user information. In oneembodiment, the system can reduce possible location types and/or useractivities as a function of the past location type and/or user activityof the positional destination.

As an example, at a given Location A, users previously studied, talked,ate, and drank. Thus, if a user's positional destination is detected asat Location A, then plausible activities postulated can be studying,talking, eating, and drinking. More so, if the given user's communitymembers only previously talked, ate, and drank, it is with a higherprobability to postulate that the given user is talking, eating, anddrinking rather than studying. Furthermore, if the given user visitedLocation A previously, but only talked and drank, then an even higherprobability is that the user is currently talking and drinking ratherthan eating and studying. It is within the scope of this invention topostulate some or all of the previously detected activities of a givenlocation. More so, it is within the scope of this invention to rankorder the activity suggestions according to the relevance of thepreviously visiting users to the given current user. As previouslydescribed, the system can request confirmation of suggested activitiesthrough the user's mobile device.

The system can additionally or alternatively reduce possible locationtypes and/or user activities as a function of the past location typeand/or user activity of the positional destination as a function of thetime of day. The system can rank possible location types and/or useractivities of the positional destination based upon known past timeperiods corresponding to the time of day of the current user visit. Forexample, again given Location A, if previous visiting users wererecorded to study one or more times during the intervals: 3:00-4:30 PMand 7:30-9:00 PM, and to drink one or more times during the intervals:4:00-7:00 PM and 8:30 PM 2:00 AM, then a current visiting user atLocation A at 3:15 PM is likely studying, at 4:15 PM is likely to beeither studying or drinking, and at 1:00 AM is likely to be drinking.More so, if the given user's community members only studied between3:15-4:30 PM then it is with a higher probability to postulate that thegiven user is studying rather than drinking at 4:15 PM. Furthermore, ifthe given user visited Location A previously but only studied, then aneven higher probability is that the user when at Location A is studying.It is within the scope of this invention to postulate some or all of thepreviously detected activities of a given location. More so, it iswithin the scope of this invention to rank order the activitysuggestions according to the relevance of the previously visiting usersto the given current user. As previously described, the system canrequest confirmation of suggested activities through the user's mobiledevice.

In embodiments of this invention, time context alone can be used topostulate activities. For example, if most days, a user is recorded tobe drinking coffee between 9:00-10:00 AM, then, without contradictoryinformation, a plausible activity postulate is that at 9:35, the user'sactivity is drinking coffee. Again, as previously disclosed, it iswithin the scope of this invention to rank order the postulated activitysuggestions according to the relevance of the previous users to thegiven current user and/or to obtain confirmation of suggestedactivities.

Additionally, it is also within the scope of this invention to rankorder the time postulates based on frequency of occurrence within thetime interval. This rank ordering applies to both location based andlocation independent time based postulates. For example, if in theinterval 4:00-4:30 PM, community members studied 25 times but drank 5times then, at 4:15, it is with a higher probability to postulate thatthe given user is studying rather than drinking.

In embodiments of the MPSM method and system of this invention, thesystem can search and/or use, if available, external, third partyinformation about the positional destination for postulating activitiesfor a given location. For example, third party vendors might provide,either free of charge or for a fee, activity information for a givensite. Consider a marketing website of a centralized homepage for agrocery store chain. Such websites are likely to contain addresses ofmany or all of the associated stores. Since these stores all supportshopping, an activity associated with these locations is shopping.Similar information can be derived or purchased from other sources suchas but not limited to commercial information repositories. Additionally,maps can be parsed. Given a location of a road, an activity of thatlocation is likely to be driving. Various and alternative third partyinformation gathering approaches and their incorporation into activityclassification and postulation can be incorporated into the method andsystem of this invention.

Suggested activity information, particularly but not limited toinformation obtained or derived from third party vendors, might beadditive or might be contradictory. Thus, combining or reconcilingpotential activities is needed. The use of voting schemes, biased basedon credibility of the source or on frequency, such as majority, or otherknown techniques, can be incorporated in the method and system of thisinvention. Note that differing suggested plausible activities mayadditive or may be contradictory. The use of techniques such as, but notlimited to, conflict resolution methods, ontology determination, andclustering, etc., can be incorporated to recognize potential conflictsand to expand classification is within the scope of this invention.

Additionally, the classification of plausible activities based onactivities occurring in the surrounding vicinity is likewise within thescope of this invention. For example, consider an unknown locationadjacent to two known locations, such as, but not limited to, twoneighboring stores or two neighboring beaches. For the neighboringstores, known activities might include shopping and strolling, while forthe neighboring beaches, known activities might include sunbathing andswimming. Given location proximity, it is within the scope of thisinvention to suggest a user's activity at the unknown location to beeither shopping and strolling or sunbathing and swimming, respectively.Confirmation can always be obtained for suggested activities and to biassuggested activities based on user familiarity and frequency ofoccurrence.

In embodiments of the MPSM method and system of this invention, localusers can opt to delay their notifications. That is, once a location isvisited or an activity occurs, a local user can opt to have thenotification of the location or activity be delayed by some period oftime. Delaying a notification provides the local user with the abilityto notify their community of the location visit or activity occurrence,but still provides the local user time to progress to the next locationor activity. As discussed above, users can also choose to automaticallyshare or not share photos taken with the digital picture frames of thisinvention.

Notifications can be complemented with correlations with other communitymembers. That is, both the local user and their respective community canbe automatically notified with a comparison. A comparison, for examplebut not limited to, can identify other community members havingpreviously conducted a specific activity or having visited a givenlocation previously. Comparisons are made by checking other communitymember locations and activities against those of the local user.Checking is performed via a search of the location-activity database. Ifa match exists within a specified or predetermined period of time, acomparison notification is made automatically. The period of time can bearbitrarily set or can follow some logical time quantum such as hour,day, week, etc.

Locations and activities are known by name. However, in addition to aname, locations and activities can have associated personal labels.Labeling locations and activities can detail familiarity to the locationand activity. User labels for locations can be surrogate names, forexample, “favorite city” for Chicago, can be songs or sound waves, forexample song words “my kind of town, Chicago is” for Chicago, can be apicture, for example “the Water Tower” for Chicago, can be a video, forexample “a panoramic view of the Chicago skyline” for Chicago, or anycombination of these and other multimedia tags supported by the localdevice. Similarly, user labels can exist for activities. For example,“favorite vice” for drinking wine, or it can be a song or sound wave,for example the song words “a bottle of red” for drinking wine, or itcan be a picture, for example, a wine bottle picture for drinking wine,or it can be a video, for example “a panoramic view of a vineyard” fordrinking wine, or any combination of these and other multimedia tagginglabels supported by the local device.

In embodiments of the MPSM method and system of this invention, localusers and community members can comment on their own and each other'slocations and activities. Comments can take any of the many multimediaforms provided by the local device. These include, but are not limitedto, text, sound, pictures, videos, or any combination thereof. Multiplecomments can be made by the local user, their community, or combinationthereof. In addition to stating their opinions (commenting), communitymembers can prompt for clarification. That is, by issuing “what”comments, community members request additional information on the postedlocations and activities. Additionally, user can “like” their own andeach other's locations and activities. By “liking” a location oractivity, community members express their satisfaction of theirrespective community members' presence in terms of location andactivity. Multiple community members as well as the local user can“like” a location and activity.

The MPSM method and systems of this invention can track vast data onboth the local user and their respective community members. These datacover, including but not limited to, locations, activities, and alsoindividuals both who are system users and those who are not. These datacan be stored and summarized. A summary of the local user and communitymember locations, activities, time durations involved in each of theselocations and activities, individuals who they encountered, etc., can becomputed and presented to the user. This summarization can range fromsimple statistical aggregation to advanced correlations as derived byany of the many, both individually and combined, machine learning, datamining, business analytics, and statistical techniques known in the art.

Information that can be aggregated or derived can answer, exemplary butnot limiting, questions such as: how much time a local user spent doingthings, such as, working at home, working out, walking the dog,commuting to work?; how much time a particular community member spentdoing things, such as, working at home, working out, walking the dog,commuting to work? (Note that the information derived for the communitymember is based strictly on the information that that particularcommunity member chose to share.); who are the five most commonindividuals that a particular user interacts with?; what is thelikelihood that after seeing a particular user, the given local userwould see a particular different individual?; which activities andlocations are most closely associated with each other and when are theymost likely to occur?; which three users among a given community aremost likely to visit a particular location together?

Local users can be provided with summaries of their locations, durationsat these locations, and activities at these locations. Furthermore, atthe discretion of the local user, these summaries are made available totheir community members.

The system can also generate and maintain both aggregation and derivedinformation. This information can be used to optimize suggestions toavoid obstacles, for example, but not limited to preferred routing ofcommuting path, promoted target advertising, for example but not limitedto location of nearby ice cream store for those users who frequentlyrecord “eating ice cream” as an activity, and a host of otherinformational services known in the art.

Digital Frames according to embodiments of this invention include aconversational agent module, e.g., a “chatbot,” which is a softwareapplication that mimics written and/or spoken human speech for thepurposes of simulating a conversation or interaction with one or moreviewers of the frame. The conversational agent preferably operates onnatural language processing and/or machine learning/data mining of photocontent and/or metadata, such as any or all of the determined/extractedinformation, photo features, photo content, and trends discussed above.The conversational agent processes or parses instructions, questions, orother spoken words (or written text) presented by the frame viewer orother user. The conversational agent responds, according to a complexseries of algorithms that interpret and identify what the viewer/usersaid, infers what viewer/user meant and/or wanted, and determines aseries of appropriate responses based on this information. The digitalphoto frames can include suitable AI chips, such as are known in theart, incorporated into the frame architecture andcommunicating/interacting with the central processor, to provide thecomputing capacity of the conversational agent. In one embodiment, theconversation involves multiple languages. In one embodiment, theviewer/user and frame communicate in different languages. In oneembodiment, the conversation involves the use of American Sign Language(ASL) captured by the camera incorporated into the frame.

In embodiments of the invention, the conversational agent is aconversation bot configured to provide interactive conversations withthe frame viewer(s). As generally shown in FIG. 13 , embodiments of thisinvention include an intelligent digital picture frame 220 including aconversation bot that interacts in back-and-forth verbal interactions230 with a viewer 240, such as using the microphone 227, a speaker 228(e.g., on rear of frame 220), and camera 225 of, or attached to, theframe 220. The verbal interaction can include a spoken statement orquestion 232 (either initial or responsive) from the viewer 240, and aspoken statement or question 234 (either initial or responsive) from theframe 220. In one embodiment, the communication from the viewer 240 isusing sign language (ASL) via camera 225. In one embodiment, thecommunication from the frame 220 is subtitled onto the display. Theverbal interaction 230 can be triggered by the viewer 240, a remotecommunity member 242 of the viewer 240 (see FIG. 14 ), or automaticallyby the frame 220 upon a triggering condition. Exemplary triggeringconditions include, without limitation, detecting a present or newviewer, detecting a viewer-preferred picture (e.g., a “liked” photo) onthe display, visually detecting a viewer response to a displayed photo,after a random or predetermined viewing period of a viewer, and/orcombinations thereof.

The verbal interactions can be a conversational interaction, such asincluding information and/or questions, and/or a story telling function,with both types of interactions preferably related to photo contentrelevant to the viewer. The conversational interaction can includechanging of the displayed photos to match the conversation topic, as theconversation proceeds. For example, if the conversation is about theviewer's dog, the frame may ask for some of the viewer's best memoriesof the dog. The viewer may reply that the dog loved the park, andchasing the geese at the lake. The frame can continually search forphotos matching the reply, and display relevant photos, includinginformation about the photo (e.g., “here is a picture of the dog at thepark in August 2020”). A slideshow of photos can thus be created tocoordinate with the conversation iterations/topics.

In embodiments of this invention, the conversational interaction can beused to determine who is viewing the frame. This can further be combinedwith viewer detection with a frame camera 225. Desirably the tone of theviewer's responses and/or the viewer's body language/facial expressionscan be used to adjust the conversation to an appropriate tone orsentiment. In desired embodiments, an automated greeting is generatedupon detecting a viewer, such as to initiate conversation interaction.

In the embodiment of FIG. 14 , the frame conversation can be initiatedremotely by a community member 242 for viewer 240. The frameconversation can be actuated through an application on the communitymember's mobile device 229 (e.g., smartphone). Such remote actuation canbe particularly helpful with viewers that may need assistance, such asvery young children or persons with mental shortcomings. The communitymember 242 can request an initiation 234 of general conversation for theviewer 240 (e.g., “how are you doing today?”) and/or request aninformational conversation about an area of interest to the viewer 240(e.g., “would you like to talk/hear about . . . ”). The community member242 may, for example, suspect that an interaction is needed based upon atone of a text or phone call received from the viewer 240 (e.g., if dadsounds sad, person 242 can actuate photos and conversation 234 aboutgrandkids or old friends/pets to cheer him up.). Upon actuation, theframe 220, upon detection of viewer 240, initiates conversation byspoken statement or question 234. If/when viewer 240 replies, theback-and-forth conversation continues. The community member 242 canselect a display photo remotely for the automated conversation (e.g.,instruct the frame to display and talk to dad about this selectedphoto), or simply request a conversation about an automatically selectedphoto. In one exemplary embodiment, frames can be set up in medicalassistance or childcare facilities, and initiated by central nursing orteacher stations.

In embodiments of this invention, automated conversations between theframe and viewer provide supplemental search capabilities to identifyphotographs for display, such as that contain specified one or moreindividuals, specified one or more activities, specified one or morelocations, and any or all combinations of individuals, activities,and/or locations. Searches can be automatically accomplished based ontechnologies discussed herein or otherwise known in the art techniques,such as those that rely on either or both metadata or image contentrelated to the photograph. As an example, the processor can determinekeywords from the conversation, and more particularly the viewer'sresponses, for searching. When paired with the clustering techniquesdescribed above, appropriate photos relevant to the viewer and/or theconversation or requests/replies/comments therein can be efficientlyidentified and displayed.

In one embodiment of the invention, these conversations provide storytelling functions, preferably based upon photo content and descriptions.Large (pretrained) language models, such as are known in the art, areused to automatically generate full stories about or including thedisplayed photograph. These stories are preferably guided by displayedimage content, metadata related to the photograph, voice prompts andsetting narratives provided by the interactive viewer, and any or allcombinations of image content, metadata, and/or voice prompts andsetting narratives. Referring to FIGS. 5 and 6 , the viewer 240 or thecommunity member 242 can request a story 234 developed by the frame 220about a favorite person, location, activity or any combination thereof,where the frame displays one or more photos of the requested contentduring the story. As an example, the viewer once lived in London, andthe story request is for a fictional or actual mystery occurring inLondon, while showing coordinating photos from the viewer's time inLondon (and perhaps supplemented from general photo repositoriesonline). The story can include community members, or various otherpeople (e.g., movie stars), as characters in the story, such asdepending on prompts from the viewer if received.

FIG. 15 illustrates one example of a story telling, where the viewer 240(e.g., a young girl) requests 232 a story 234 about a princess, perhapsherself as a princess. The story request 232 can be made upon seeing aphoto of herself in her princess outfit on frame 220, or the displayedphoto can be shown after, as relevant to, the story request 232. Theframe may ask for additional details, such as the princess's name or adesired adventure (e.g., meeting a prince or fighting a dragon). Theframe may also allow the viewer to dictate parts of the story, such asin a choose-your-adventure story type (e.g., “should the princess enterthe dark cave?”). The level of continued interaction can preferably beset at the beginning of the story by the viewer, so as to minimize storydisruption if desired. The conversation and storytelling features ofthis invention can also be implemented through the frame's interactionapplication on a user's mobile device, such as to provide thesephoto-based stories as entertainment for a child when away from theframe (e.g., a car ride).

FIG. 16 shows the photo of FIG. 5 on display 220, and where a viewerrequests a pirate treasure story. As shown in FIGS. 17 and 18 , thephoto can be annotated with overlay images 250 and/or sounds 252, suchas a discovered treasure chest and seagulls/ocean waves in FIG. 17 . Thestory continues in FIG. 18 , where pirates return to look for thetreasure that the persons in the photo discovered, etc. Overlay images250 and sounds 252 can be obtained from various online databases orotherwise stored in the frame system.

In embodiments, the conversation or story occurs in or incorporates oneor more non-native (foreign) language(s). In one embodiment theconversation occurs with a given dialect (British vs. American English)and/or a given accent (Northern Germany vs. Bavarian pronunciation). Forexample, character dialogue in the London mystery above can be presentedin an English accent. Different voices for different characters is alsocontemplated. The conversation can also be text-based, e.g.,close-captioned, as needed. The frame can also be used for foreignlanguage lessons, pairing the photos with non-native equivalent words(e.g., “Papa and ich am Strand” or “

” for FIG. 16 ). In one embodiment, multiple scripts are supported. Inone embodiment the captioning provides either both or just thetransliteration of the response such as “Aba ve'ani al hakhof” and/or “

”.

In embodiments of storytelling, animation is provided. In embodiments ofstorytelling, image composition for panoramic viewing of a scene isprovided. In embodiments of storytelling, mood is intensified bybackground music selection; the music can be obtained from frameresident or external sources or self-generated via large languagemodels, potentially guided to generate a particular tone, style, genre,or mood.

In embodiments, the conversation or story-telling interaction occurswith virtual bots representing the people in the photograph displayed.In one embodiment the intended or desired sentiment of the conversationis provided by the viewer. In one embodiment the motivation or purposeof the conversation is provided by the viewer. For example, it ispossible that the motivation is to serve an educational mission(potentially is the case for describing an activity) or to providecomfort to the viewer (in the case of recent loss). In embodiments, theconversation is governed by viewer demographics such as but not limitedto age or gender. Sentiment analysis is a subfield of computer sciencethat uses NLP and machine learning to measure the sentiment and tone ofa text or spoken language. Sentiment analysis can help a chatbot analyzeuser messages and identify whether the person's attitude towards certainproducts or services is negative, positive, or neutral.

To assist the intelligent conversation or story-telling interactions,the invention includes a method and system that extracts one or more ofcontent features from photos of the digital photo collection(s), and/orphoto image features from (same or different) photos of the digitalphoto collection(s), as described above. As an example, FIG. 16 shows aphoto of a man and young girl at the beach. The photo can be tagged asdescribed above for FIG. 5 as labeled based upon the colors, shapes,light intensity, and/or color histogram of the photo.

It is within the scope of this invention to again limit the taggingoptions. It is also within the scope to conflate multiple tag optionsinto fewer and/or of fixed vocabulary. In embodiments of this invention,the photo extraction and/or analysis is limited to look forpredetermined desired information or features that may be particularlyuseful for conversation. For example, the photo extraction or analysiscan be limited to a particular person and/or location (e.g., tropics orspring break) that may be desired by a particular viewer.

In embodiments of this invention, the limits of tagging options can bebased on the viewer's identity, as potentially detected via facerecognition from the photo captured by camera 225. In embodiments ofthis invention, the limits of tagging options can be based on theviewer's characteristics such as but not limited to age and gender, aspotentially detected via age or gender recognition software based on thephoto captured by camera 225. In embodiments of this invention, thelimits of tagging options can be based on parental control software tolimit access of a viewer based on identity or age using the photocaptured by camera 225.

In embodiments of this invention the information detected in photos canbe clustered and can be used for conversation and story-tellingpurposes. The metadata clustering process for trend determination can besimilar to that discussed above for frame display, however, is generallyfor a different purpose and using different criteria/photo elements, andcan be used in addition to the clustering described herein for framedisplay purposes. As an example, tagged photos can be organized into aplurality of sub-clusters, each for a corresponding common detectedextracted content feature and/or extracted photo image feature, such asby activity, location, temperature/weather/seasons, time, attributes ofa person of the photos, light intensity of the photo, or a color withinthe photos, etc., and various combinations of these categories.Correlations between photos can be mined, such as by comparingsub-clusters of photos across more than one digital photo collectionover a network. Example correlations include determining common people,locations, and/or activities within the tagged photos, such as for apredetermined time period and/or a predetermined family group.

Thus, the invention provides a digital picture frame including a camera,microphone, and speaker connected to the frame, and a network connectionmodule for use as a device for displaying pictures from a user'selectronic device and/or social media account or her or his communitymembers' social media accounts. The frame allows for efficient,automated access to photos relevant to the viewer(s) of the frame, aswell as conversational interactions with a viewer. The automated frameallows for changing photos for the viewer(s) without multiple manualsteps.

The invention illustratively disclosed herein suitably may be practicedin the absence of any element, part, step, component, or ingredientwhich is not specifically disclosed herein.

While in the foregoing detailed description this invention has beendescribed in relation to certain preferred embodiments thereof, and manydetails have been set forth for purposes of illustration, it will beapparent to those skilled in the art that the invention is susceptibleto additional embodiments and that certain of the details describedherein can be varied considerably without departing from the basicprinciples of the invention.

What is claimed is:
 1. A digital picture display system, comprising: adigital photo frame including a digital display, a network connectionmodule adapted to connect to a user electronic device and to receivephotos stored within the user electronic device, an automated displaymodule adapted to automatically display photos and change photosdisplayed on the digital display from the user electronic device, amicrophone in combination with the automated display module and adaptedto receive spoken communications, and a speaker in combination with theautomated display module and adapted to produce audio communications; aninteraction application stored on the user electronic device, the userelectronic device including a digital photo collection, and theinteraction application including a plurality of automatically createddigital photo clusters according to photo content from the digital photocollection; the digital photo frame further including a conversationalagent module configured to provide an automated verbal interaction witha viewer of the digital picture frame, wherein the conversational agentmodule determines a search parameter from the verbal interaction andinitiates a search of the digital photo clusters for metadata tags anddigital photos matching the search parameter, wherein the networkconnection module receives matching digital photos from one or more ofthe digital photo clusters for display on the digital display; whereinthe conversational agent module includes a story-telling moduleconfigured to receive a story request from the viewer of the digitalpicture frame, automatically search for and/or display one or moredigital photos of the digital photo collection matching the storyrequest, and automatically generate a unique narrative through thespeaker matching the story request and the one or more digital photos,wherein the unique narrative is guided by displayed image content, photometadata, or settings provided by the viewer; and wherein theconversational agent module is configured to automatically detect asentiment or mood of the viewer from the verbal interaction, and toautomatically adjust a storyline of the unique narrative in response tothe verbal interaction, and the detected sentiment or mood.
 2. Thesystem of claim 1, wherein the conversational agent module is configuredto automatically determine with questions through the speaker anidentity of the viewer of the digital picture frame.
 3. The system ofclaim 1, wherein the verbal interaction comprises a plurality ofautomated back-and-forth iterations of conversational vocal utterancesvia the microphone and speaker to establish an identity of the userand/or the search parameter.
 4. The system of claim 1, wherein thesearch parameter includes a person, an activity, a location, andcombinations thereof, identified from keywords detected in the verbalinteraction.
 5. The system of claim 1, wherein the conversational agentmodule receives the story request in response to a photo being displayedon the digital display.
 6. The system of claim 1, wherein theconversational agent module instructs the automated display module toshow a slideshow of photos related to the unique narrative.
 7. Thesystem of claim 1, wherein the unique narrative is developed fromcontent clustered photographs of the digital photo collection, clusteredaccording to a person, an activity, a location, or combinations thereof,and/or developed from metadata associated with photos in the digitalphoto collection.
 8. The system of claim 1, wherein the search isperformed on content clustered photographs of the digital photoclusters, clustered according to a person, an activity, a location, orcombinations thereof, and/or performed on metadata associated withphotos in the digital photo collection.
 9. The system of claim 1,wherein the interaction application stores extracted content featurescomprising locations, seasons, weather content, times of day, dates,activity content, and/or one or more attributes of persons within thephotos.
 10. The system of claim 1, wherein predetermined relationshipsbetween different metadata tags of a plurality of tagged photos are usedin the verbal interaction.
 11. The system of claim 10, wherein theinteraction application and/or the conversational agent module isconfigured to mine correlations within the metadata tags.
 12. The systemof claim 11, wherein the mined correlations comprises determined popularlocations, clothing items, and/or activities within the plurality of thetagged photos, for use in the verbal interaction.
 13. The system ofclaim 11, wherein the verbal interaction comprises a plurality ofautomated back-and-forth conversational iterations, and the minedcorrelations comprise determined popular locations, clothing items,and/or activities within a plurality of tagged photos, for use ascontent in conversational iterations of the verbal interaction.
 14. Thesystem of claim 1, wherein the conversational agent module is configuredto detect and analyze responsive vocal utterances or voice prompts fromthe viewer during the verbal interaction to automatically adjust theverbal interaction or change a displayed photo in response to theresponsive vocal utterances or voice prompts.
 15. The system of claim14, wherein a further photo is obtained from the digital photocollection and the conversational agent module is configured toautomatically continue the verbal interaction with a viewer of thedigital picture frame about the displayed second photo.
 16. The systemof claim 1, wherein the verbal interaction includes automaticallygenerated questions to the viewer about a displayed photo.
 17. Thesystem of claim 1, further comprising a second interaction applicationstored on a second electronic device of a community member, the secondelectronic device including a second digital photo collection, and thesecond interaction application including a second plurality ofautomatically created digital photo clusters according to content fromthe second digital photo collection, wherein the search parameter fromthe verbal interaction is searched within the digital photo clusters ofthe second digital photo collection for metadata tags and digital photosmatching the search parameter, wherein the network connection modulereceives matching digital photos from the second digital photocollection for display on the digital display.
 18. The system of claim1, further comprising a facial recognition module in combination withthe automated display module, wherein the automated display moduleautomatically changes a photo display upon detection of a recognizedviewer with the facial recognition module.
 19. The system of claim 18,wherein the automated display module includes or obtains a viewerprofile of the recognized viewer and automatically changes the displayas a function of the viewer profile and the corresponding electronicdevice upon detection of the recognized viewer through the camera.